RESOLVED: Fusion 360 Service Outage - 9/10/18

[brianrepp]

All - as most of you have hopefully already seen either through the in-product notification, or our Health Dashboard, we had to take Fusion 360 offline due to a service issue that started around 11:30am PT.

Teams have been actively working to resolve the issue, but I do not have an ETA at this time.  As soon as I have more information, I will update this forum thread, along with our Health Dashboard.

I understand this is disruptive, and appreciate your continued patience as we work to get you back up and running.

UPDATE as of 10:00pm PT - Thank you for your continued patience!  We have resolved the issue and are taking Fusion 360 back online now.  Please note that it may take up to 20 minutes for your session to reconnect.  If you run into any delays or would like to speed up the process, you can also restart Fusion 360.  Please note, as we go back online, there may be some delays in processing due to the backlog.

What happened & how will we prevent it in the future?  

The recent issue with our Access Control Management service impacted customers’ access to certain workflows within several Autodesk applications. We understand these issues are disruptive to your business and apologize for any inconvenience.  All services were fully operational by 11:30pm PT, September 10 and we do not expect any further disruptions related to this issue. 

What happened: We experienced an issue with the capacity scaling limit configuration for our Access Control Management database. This database manages access control policies for data stored by Autodesk products.  This was not a security issue. 

  

Mitigation: The issue was fully resolved with a code fix and we are reviewing all our production databases for any similar issues as a preemptive measure to avoid any further customer disruptions. 

[brianrepp]

All - as most of you have hopefully already seen either through the in-product notification, or our Health Dashboard, we had to take Fusion 360 offline due to a service issue that started around 11:30am PT.

Teams have been actively working to resolve the issue, but I do not have an ETA at this time.  As soon as I have more information, I will update this forum thread, along with our Health Dashboard.

I understand this is disruptive, and appreciate your continued patience as we work to get you back up and running.

UPDATE as of 10:00pm PT - Thank you for your continued patience!  We have resolved the issue and are taking Fusion 360 back online now.  Please note that it may take up to 20 minutes for your session to reconnect.  If you run into any delays or would like to speed up the process, you can also restart Fusion 360.  Please note, as we go back online, there may be some delays in processing due to the backlog.

What happened & how will we prevent it in the future?  

The recent issue with our Access Control Management service impacted customers’ access to certain workflows within several Autodesk applications. We understand these issues are disruptive to your business and apologize for any inconvenience.  All services were fully operational by 11:30pm PT, September 10 and we do not expect any further disruptions related to this issue. 

What happened: We experienced an issue with the capacity scaling limit configuration for our Access Control Management database. This database manages access control policies for data stored by Autodesk products.  This was not a security issue. 

  

Mitigation: The issue was fully resolved with a code fix and we are reviewing all our production databases for any similar issues as a preemptive measure to avoid any further customer disruptions. 

[brianrepp]

Update as of 2:45pm PT, we have found the root cause of the issue and are working to restore the impacted services.  I will report back by 4pm PT with an update on the status.

[brianrepp]

Update as of 2:45pm PT, we have found the root cause of the issue and are working to restore the impacted services.  I will report back by 4pm PT with an update on the status.

[bigbrad73]

somehow you guys have to get these outages figured out! wouldn't you know I had a big review at a huge customer and sure enough fusion is offline! I like fusion more and more everyday until each time this happens. Is this kind of stuff happening because the updates are rushed? I know sometimes things go wrong, I do...but always seems like this happens on days like today when you need it the most.

[bigbrad73]

somehow you guys have to get these outages figured out! wouldn't you know I had a big review at a huge customer and sure enough fusion is offline! I like fusion more and more everyday until each time this happens. Is this kind of stuff happening because the updates are rushed? I know sometimes things go wrong, I do...but always seems like this happens on days like today when you need it the most.

[quade.hale]

4pm PDT has passed. I've got STLs to export and it's is broken completely (it has worked offline before but apparently an outage is a different ballgame). Do you have an ETA?

[quade.hale]

4pm PDT has passed. I've got STLs to export and it's is broken completely (it has worked offline before but apparently an outage is a different ballgame). Do you have an ETA?

[brianrepp]

---

@bigbrad73 wrote:

somehow you guys have to get these outages figured out! wouldn't you know I had a big review at a huge customer and sure enough fusion is offline! I like fusion more and more everyday until each time this happens. Is this kind of stuff happening because the updates are rushed? I know sometimes things go wrong, I do...but always seems like this happens on days like today when you need it the most.

---

@bigbrad73 - I feel like the only time you and I talk are during outages, we have to stop meeting like this!    

As you can imagine, Fusion 360 leverages a number of different services that are shared across various products.  The significant majority of these unplanned outages (today included) are not tied to any one version update of Fusion 360, or the testing of those updates, but instead the services behind the scenes.  In recent years especially, those services have become incredibly stable thanks to the work of our teams, but things still happen, and with each hiccup we learn and do everything we can to prevent similar future problems.

While things have become a lot more stable, and F360 has better handling of these service issues, we also recognize there is still work to be done and it is a priority for us.  It has a significant impact on users like yourself, and we don't take that lightly.

[brianrepp]

---

@bigbrad73 wrote:

somehow you guys have to get these outages figured out! wouldn't you know I had a big review at a huge customer and sure enough fusion is offline! I like fusion more and more everyday until each time this happens. Is this kind of stuff happening because the updates are rushed? I know sometimes things go wrong, I do...but always seems like this happens on days like today when you need it the most.

---

@bigbrad73 - I feel like the only time you and I talk are during outages, we have to stop meeting like this!    

As you can imagine, Fusion 360 leverages a number of different services that are shared across various products.  The significant majority of these unplanned outages (today included) are not tied to any one version update of Fusion 360, or the testing of those updates, but instead the services behind the scenes.  In recent years especially, those services have become incredibly stable thanks to the work of our teams, but things still happen, and with each hiccup we learn and do everything we can to prevent similar future problems.

While things have become a lot more stable, and F360 has better handling of these service issues, we also recognize there is still work to be done and it is a priority for us.  It has a significant impact on users like yourself, and we don't take that lightly.

[brianrepp]

@quade.hale - apologies i'm a few minutes late, I was getting an update on the situation and also responding to Brad's earlier post.

Update 4:08pm PT - the teams are getting close to finalizing the fix and will then be working through testing before releasing.  I still do not have an ETA, but I would expect this could still take up to a few hours to make sure everything works as expected before releasing to the wild.

[brianrepp]

@quade.hale - apologies i'm a few minutes late, I was getting an update on the situation and also responding to Brad's earlier post.

Update 4:08pm PT - the teams are getting close to finalizing the fix and will then be working through testing before releasing.  I still do not have an ETA, but I would expect this could still take up to a few hours to make sure everything works as expected before releasing to the wild.

[quade.hale]

Fair enough, thanks for the update.

[quade.hale]

Fair enough, thanks for the update.

[colinNJB25]

Interestingly I had cached my drawing templates as they are the files I use the most but even though I can open them offline I can't include them into a new drawing.

[colinNJB25]

Interestingly I had cached my drawing templates as they are the files I use the most but even though I can open them offline I can't include them into a new drawing.

[Anonymous]

How often do these outages occur?

[Anonymous]

How often do these outages occur?

[brianrepp]

@Anonymous - you can take a look back at our history of events on the Health Dashboard (click on the History tab).

[brianrepp]

@Anonymous - you can take a look back at our history of events on the Health Dashboard (click on the History tab).

[brianrepp]

---

@colinNJB25 wrote:

Interestingly I had cached my drawing templates as they are the files I use the most but even though I can open them offline I can't include them into a new drawing.

---

Apologies for the delay responding here, but I wanted to confirm with the Drawings team if this is expected behavior.  They confirmed it is as-designed today, but they'll look for ways to give access to these when offline.  Thank you for raising this!

[brianrepp]

---

@colinNJB25 wrote:

Interestingly I had cached my drawing templates as they are the files I use the most but even though I can open them offline I can't include them into a new drawing.

---

Apologies for the delay responding here, but I wanted to confirm with the Drawings team if this is expected behavior.  They confirmed it is as-designed today, but they'll look for ways to give access to these when offline.  Thank you for raising this!

[richard]

This response really isn't good enough.

You've intentionally designed Fusion 360 so that it cannot store design files locally--meaning that the tool is almost entirely useless when cloud storage is not accessible.  And then you repeatedly demonstrate that the cloud connection provided by Autodesk is not reliable.  The result is that your whole product becomes unusable on a periodic basis.

With such a blatant flaw in your product's structure, I struggle to understand why anyone would use this product for anything except hobby/student work.  If I had a multi-million-dollar business involved I would drop your product like it had the plague, and invest instead in a product that I can use without being continually dependent upon your IT department.

[richard]

This response really isn't good enough.

You've intentionally designed Fusion 360 so that it cannot store design files locally--meaning that the tool is almost entirely useless when cloud storage is not accessible.  And then you repeatedly demonstrate that the cloud connection provided by Autodesk is not reliable.  The result is that your whole product becomes unusable on a periodic basis.

With such a blatant flaw in your product's structure, I struggle to understand why anyone would use this product for anything except hobby/student work.  If I had a multi-million-dollar business involved I would drop your product like it had the plague, and invest instead in a product that I can use without being continually dependent upon your IT department.

[jasondentlerautodesk]

Thanks for the update.

To export my STLs, I had to export the f3d, open the f3d file, go offline, and then it allowed me to export my STLs. This seems like something you should address once the outage is over.

Also, I can't render locally due to the outage because my file is "unsaved" - not synced up to the cloud.

[jasondentlerautodesk]

Thanks for the update.

To export my STLs, I had to export the f3d, open the f3d file, go offline, and then it allowed me to export my STLs. This seems like something you should address once the outage is over.

Also, I can't render locally due to the outage because my file is "unsaved" - not synced up to the cloud.

[engineeringNCMXB]

After having various problems with cloud services and hard crashes all through the weekend....now this. 

Right now....I cannot even start Fusion 360. All day. The estimated cost for losing that last few days is more than a standalone license of Solidworks (which I already have 20 years with). In those 2 decades, I have never lost a whole day to technical issues - not even a few hours. Maybe a 15 to 20 hard crashes during that time and no lost data. It just worked. Fusion has eaten the last 3 days - dwarfing the total lost time with Solidworks over the course of 20 years. Nuts. 

Fustion 360 is 'almost' amazing. Lots of promise and good ideas.....but as far as features go the most important possible feature is RELIABILITY. It is the tip top feature that every other function must be second.  

[engineeringNCMXB]

After having various problems with cloud services and hard crashes all through the weekend....now this. 

Right now....I cannot even start Fusion 360. All day. The estimated cost for losing that last few days is more than a standalone license of Solidworks (which I already have 20 years with). In those 2 decades, I have never lost a whole day to technical issues - not even a few hours. Maybe a 15 to 20 hard crashes during that time and no lost data. It just worked. Fusion has eaten the last 3 days - dwarfing the total lost time with Solidworks over the course of 20 years. Nuts. 

Fustion 360 is 'almost' amazing. Lots of promise and good ideas.....but as far as features go the most important possible feature is RELIABILITY. It is the tip top feature that every other function must be second.  

[brianrepp]

Good feedback @jasondentlerautodesk - i've captured and will pass along to the appropriate teams.

[brianrepp]

Good feedback @jasondentlerautodesk - i've captured and will pass along to the appropriate teams.

[brianrepp]

---

@engineeringNCMXB wrote:

After having various problems with cloud services and hard crashes all through the weekend....now this. 

Right now....I cannot even start Fusion 360. All day. The estimated cost for losing that last few days is more than a standalone license of Solidworks (which I already have 20 years with). In those 2 decades, I have never lost a whole day to technical issues - not even a few hours. Maybe a 15 to 20 hard crashes during that time and no lost data. It just worked. Fusion has eaten the last 3 days - dwarfing the total lost time with Solidworks over the course of 20 years. Nuts. 

Fustion 360 is 'almost' amazing. Lots of promise and good ideas.....but as far as features go the most important possible feature is RELIABILITY. It is the tip top feature that every other function must be second.  

---

@engineeringNCMXB - I couldn't agree more, and we understand without reliable access nothing else matters.  This outage in particular has been painful.  We're doing everything we can to get back online as quickly as possible.

I'm not aware of any cloud service issues over the weekend, so would like to better understand what happened.  If you have time, would you mind sending me an email to brian dot repp at autodesk.com describing the issue (both with the services, and the crashes)?

[brianrepp]

---

@engineeringNCMXB wrote:

After having various problems with cloud services and hard crashes all through the weekend....now this. 

Right now....I cannot even start Fusion 360. All day. The estimated cost for losing that last few days is more than a standalone license of Solidworks (which I already have 20 years with). In those 2 decades, I have never lost a whole day to technical issues - not even a few hours. Maybe a 15 to 20 hard crashes during that time and no lost data. It just worked. Fusion has eaten the last 3 days - dwarfing the total lost time with Solidworks over the course of 20 years. Nuts. 

Fustion 360 is 'almost' amazing. Lots of promise and good ideas.....but as far as features go the most important possible feature is RELIABILITY. It is the tip top feature that every other function must be second.  

---

@engineeringNCMXB - I couldn't agree more, and we understand without reliable access nothing else matters.  This outage in particular has been painful.  We're doing everything we can to get back online as quickly as possible.

I'm not aware of any cloud service issues over the weekend, so would like to better understand what happened.  If you have time, would you mind sending me an email to brian dot repp at autodesk.com describing the issue (both with the services, and the crashes)?

[brianrepp]

Update 7:30pm PT - it's looking promising, but work still being done along with significant testing before deploying the fix. While I still don't have a firm ETA, it could still be a few more hours.

[brianrepp]

Update 7:30pm PT - it's looking promising, but work still being done along with significant testing before deploying the fix. While I still don't have a firm ETA, it could still be a few more hours.

[bradmuise]

WTF.....I heard there was a 24hr outage in 2016....see the youtube video below....this is BS....

https://www.youtube.com/watch?v=lXf5EeO6ePs

[bradmuise]

WTF.....I heard there was a 24hr outage in 2016....see the youtube video below....this is BS....

https://www.youtube.com/watch?v=lXf5EeO6ePs

[brianrepp]

@bradmuise - Indeed, and it was painful for all.  We've come a LONG way since then (both in our services, and offline experience), but nevertheless today's outage is also rough and i'm hopeful we can be back online soon enough.

[brianrepp]

@bradmuise - Indeed, and it was painful for all.  We've come a LONG way since then (both in our services, and offline experience), but nevertheless today's outage is also rough and i'm hopeful we can be back online soon enough.

[Anonymous]

Hi Brian,

Thanks for the updates, but I have to echo richard's comments above.

As a systems and networking engineer, I'm already pretty skeptical of the push to cloud services is anything more than a cash grab to begin with. Just admitting my bias up-front.

So, while I'm just learning CAD for the first time with Fusion 360 for my new 3D printing hobby, I have to agree that if I made a business out of this that could afford a Solidworks license, I would switch in a heartbeat just for the reason that it works perfectly without being connected to the cloud.

I've been waiting all day to be able to work on my first project and been unable to import new components due to the difficulties. If I depended on F360 for a business, this would not be acceptable. Synchronizing data to the cloud as a method of backing up data I can understand. Making a cloud connection mandatory for getting work done is just foolish.

[Anonymous]

Hi Brian,

Thanks for the updates, but I have to echo richard's comments above.

As a systems and networking engineer, I'm already pretty skeptical of the push to cloud services is anything more than a cash grab to begin with. Just admitting my bias up-front.

So, while I'm just learning CAD for the first time with Fusion 360 for my new 3D printing hobby, I have to agree that if I made a business out of this that could afford a Solidworks license, I would switch in a heartbeat just for the reason that it works perfectly without being connected to the cloud.

I've been waiting all day to be able to work on my first project and been unable to import new components due to the difficulties. If I depended on F360 for a business, this would not be acceptable. Synchronizing data to the cloud as a method of backing up data I can understand. Making a cloud connection mandatory for getting work done is just foolish.