RESOLVED: Fusion 360 Service Outage

[keqingsong]

UPDATE DEC 8, 2;22 A.M. PST -

All, tests on our end have shown that we are back online. You should now be able to log in and access your data and use Fusion 360 as you normally would. Things are still ramping up, but if you continue notice any sluggishness or issues, please let us know. Thank you for your patience, sympathy, words of encouragement, and continued support. It meant a lot to all of us!  

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 9:17 P.M. PST -

All – A quick update on the outage.  We are close to a resolution.  We have started testing Fusion in our internal staging environment.  Once we are sure everything works, we will go live and bring Fusion online. I know this has been very painful for all of you and all of us in the fusion team feel your pain.  I really appreciate your patience as we work the problem.  Once we get thru this and you are all up and running, I will update you on the work we are doing and are doubling down to make Fusion resilient and more importantly make sure that you have access to your data all the time.

In case you are wondering what the current issue is why it is taking so long to address it, here are the details.  Fusion 360 is a network of services and one such service is what we call ACM (Access Control Manager) Service.  The purpose of ACM is to govern project level access to all data resources (items, projects, folders…) and its derivatives (views, comments, …).  We periodically optimize our Fusion Item Access Control Database, based on changing application needs and usage of ACM.  An optimization done on Dec 6 made calls to ACM getting locked up in the database and never return. Ultimately, we had to bring our ACM service down to roll back the changes and make sure all the services operate normally.  It is an involved effort and hence the delay.

Prabakar.

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 11:05 A.M. PST - We are now estimating to have the outage resolved by 1pm PST.  I will keep this thread updated if anything changes.  Thanks for your continued patience.

UPDATE DEC 7, 7:19 A.M. PST - We are making progress on battling the outage. We have been having issues with our Access Control System that governs project level access to all data resources (items, projects, folders...) and its derivatives (views, comments, ...) in Fusion for creators and collaborators.

Our Operations and Engineering Team is engaged ferociously since midnight Dec 6 to address this issue. We expect a speedy resolution of this issue soon.

Right now we just deployed a banner in-product that will take you into offline mode if you haven't already done so. More details to come, and again apologies for the downtime.

Hey all, 

We are experiencing an outage that is impacting all data access interactions.  These include but not limited to Open, Save, Upload, Export and Viewing. We recommend that you work in offline mode for the time being. We are working actively to correct the problem and will update this thread once it is resolved.  We apologize for the inconvenience and appreciate your patience.

You can also keep track of the status via the health dashboard. 

Here's how to switch to offline mode to continue working while we dig into the issue. 

https://knowledge.autodesk.com/support/fusion-360/troubleshooting/caas/sfdcarticles/sfdcarticles/How...

If you restarted Fusion without going into offline mode before, it may seem like fusion is hanging at the launch screen. Wait a few minutes and you should be able to get in. As soon as you get in, switch to offline mode. Then you should be good to go while we fix the downtime.

[neljoshua]

First, thanks to all Fusion team members for working on this.  I am sure things are quite hectic for you all right now.

Second, I looked around on the IdeaStation for posts that dealt with the issue of automatic local storage.  I found three (here, here, and here), but they have all been archived, so I created a new one.  It can be found here.

Please vote if you like the idea!

[brianrepp]

UPDATE - We are now estimating to have the outage resolved by 1pm PST.  I will keep this thread updated if anything changes.  Thanks for your continued patience...

[NYC_CNC]

Thanks for sharing that, Brian.  I know it isn't a guarantee, but it's so helpful to hear SOME estimate.   It makes us feel like we're working with 'real people' and not some faceless technology company.   Thank you.

[GMDesigns74]

Access to your own data on the cloud isn't the only issue - my sporadic need for Fusion likely led to this, but I haven't seen this one mentioned in the thread:

Thus I have no option to enter into offline mode (clicking ok exits the application)

[ARTHUR-HM]

We have two of our users here at our shop with this error. We have a bunch of issues as newer files weren't sunk to everyone's machine yet. So we have parts that need to get made ASAP and having a hard time searching who has access to what and which ones are the latest versions. It's a mess. Not happy at all about this.

[EdDRM]

Hello,

 I cannot find the offline cache preference you metioned.

What is it under?

Thank you.

Found it..it's set to 60 days.

Where do I find the work I was doing yesterday?

[Anonymous]

@EdDRM If you open the Data Panel, all of your cached files will be listed. As far as I can tell, they are in no particular order.

[EdDRM]

jason.tebbe

My data panel was empty, I restarted Fusion and now it shows up.

Thanks for the response.

[etfrench]

Not necessarily, the file I was working on last night when the connection failed is not in my data panel.  With luck it may still exist in the cloud.

[brianrepp]

UPDATE - based on current progress, the new ETA for getting this resolved is 6pm PST.  Thanks for sticking with us.

[mavigogun]

The ever-moving goal line erodes credibility.   I'm a bit torn as to how to handle this- each prediction undermines the next.   Tough spot.   Glad it ain't me.

[sambirchenough]

Thanks @brianrepp. I echo John here, the situation sucks but the human updates are much appreciated. I know you guys are working hard.

[Anonymous]

---

@neljoshua wrote:

First, thanks to all Fusion team members for working on this.  I am sure things are quite hectic for you all right now.

Second, I looked around on the IdeaStation for posts that dealt with the issue of automatic local storage.  I found three (here, here, and here), but they have all been archived, so I created a new one.  It can be found here.

Please vote if you like the idea!

---

Thanks for putting that up.  That's exactly what I want - everything stored locally for as long as I want, with "cloud" as a nice bonus when it's working.

I very much appreciate the hard work the team is putting in to fix this problem, but it inevitably reveals why this approach means that Fusion 360 can't be a critical tool, and why, as much as I tell colleagues how much I like it, what a great value it is at current pricing, they have to be wary of the fact that it can be non-functional at any time.

[Brad_Southard]

Thank you Brian, I do appreciate the updates, even if the deadline does continue to be pushed.   The work I had planned for today, has not been able to happen today, for some reason the file I was working on last night is not available in the offline mode Cache.  

This has however made it clear to me that I need a backup of all my design data from the cloud, on a local server, or computer.   Is there a method to bulk download all of those files, or a best practice to download them individually?  

Thanks much 

Brad

[SGL-Design]

I'm sure you are extremely busy, but I'm interested in hearing the proposed solutions for this problem. Will you still be sharing those today?

[EdDRM]

Josh.nelson,

I voted for your idea, hopefully everyone takes the time to do so and maybe we will see some change.
What is happening today is costing a lot of people a lot of money$$$ and at some point,

no matter how cheap a product is it just is not going to be worth it.

I do want to thank everyone at Autodesk that is and has been working their a$$ off to fix this as they are not the problem.

[charegb]

@GMDesigns74 I think I have a work around for this error.

1) Close Fusion

2) Go to %localappdata%\Autodesk in windows explorer

3) Rename the "Web Services" folder to something like "Web Services backup"

4) Launch Fusion - It will prompt you to login and then show you all the cached data in offline mode.

For any Mac users reading this, the location is - /Users/$username/Library/Application Support/Autodesk

Let us know if this works.

Thanks,

Bankim

[scottmoyse]

---

@Anonymous wrote:

---

@neljoshua wrote:

First, thanks to all Fusion team members for working on this.  I am sure things are quite hectic for you all right now.

Second, I looked around on the IdeaStation for posts that dealt with the issue of automatic local storage.  I found three (here, here, and here), but they have all been archived, so I created a new one.  It can be found here.

Please vote if you like the idea!

---

Thanks for putting that up.  That's exactly what I want - everything stored locally for as long as I want, with "cloud" as a nice bonus when it's working.

I very much appreciate the hard work the team is putting in to fix this problem, but it inevitably reveals why this approach means that Fusion 360 can't be a critical tool, and why, as much as I tell colleagues how much I like it, what a great value it is at current pricing, they have to be wary of the fact that it can be non-functional at any time.

---

The fact that all those ideas have been archived just goes to show that this access issue hasn't been taken seriously enough. From the very first day of Fusion 360's commercial existence, the number one priority should have been making sure that the end user has access to their data 100% of the time no matter the circumstances! It doesn't matter what anyone from Autodesk says retrospectively about this, the fact is it wasn't and apparently still isn't the number 1 priority! An additional complication has become apparent today, if you are using Fusion team and switch hubs, when in offline mode, you don't have access to the recently accessed files from any of your other hubs. I'm not clear at this point if switching hubs clears your cache, or just switches to a different cache. One of our customers just switched over to using their Team hub yesterday, so all the files they were working on prior to that aren't accessible to them now until the servers come back up. Their whole shop is down now, because the scheduled work for the next few days isn't accessible to them. This is a fundamental issue with Fusion Team that needs to be addressed as well. Good luck with getting this sorted before your next estimated time.

[scottmoyse]

---

@Brad_Southard wrote:

or some reason the file I was working on last night is not available in the offline mode Cache.  

---

There needs to be MUCH better clarity surrounding the conditions/requirements that leads to files being cached. We have another customer in the same boat as you. They were working on stuff yesterday, and not all of it has been cached, but older files are there.

[rodwyll]

For everyone requesting a local storage system: they already announced this in the product innovation keynote at Autodesk university 2016:

There will be a file system similar to a dropbox folder on your computer, you can put files there in your file system and it will automatically synced to the cloud. 

This cloud failure is incredibly frustrating and Im glad I dont have any project deadlines coming up, but Im sure they will implement safeguarding systems to prevent this from happening again.