RESOLVED: Fusion 360 Service Outage

[keqingsong]

UPDATE DEC 8, 2;22 A.M. PST -

All, tests on our end have shown that we are back online. You should now be able to log in and access your data and use Fusion 360 as you normally would. Things are still ramping up, but if you continue notice any sluggishness or issues, please let us know. Thank you for your patience, sympathy, words of encouragement, and continued support. It meant a lot to all of us!  

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 9:17 P.M. PST -

All – A quick update on the outage.  We are close to a resolution.  We have started testing Fusion in our internal staging environment.  Once we are sure everything works, we will go live and bring Fusion online. I know this has been very painful for all of you and all of us in the fusion team feel your pain.  I really appreciate your patience as we work the problem.  Once we get thru this and you are all up and running, I will update you on the work we are doing and are doubling down to make Fusion resilient and more importantly make sure that you have access to your data all the time.

In case you are wondering what the current issue is why it is taking so long to address it, here are the details.  Fusion 360 is a network of services and one such service is what we call ACM (Access Control Manager) Service.  The purpose of ACM is to govern project level access to all data resources (items, projects, folders…) and its derivatives (views, comments, …).  We periodically optimize our Fusion Item Access Control Database, based on changing application needs and usage of ACM.  An optimization done on Dec 6 made calls to ACM getting locked up in the database and never return. Ultimately, we had to bring our ACM service down to roll back the changes and make sure all the services operate normally.  It is an involved effort and hence the delay.

Prabakar.

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 11:05 A.M. PST - We are now estimating to have the outage resolved by 1pm PST.  I will keep this thread updated if anything changes.  Thanks for your continued patience.

UPDATE DEC 7, 7:19 A.M. PST - We are making progress on battling the outage. We have been having issues with our Access Control System that governs project level access to all data resources (items, projects, folders...) and its derivatives (views, comments, ...) in Fusion for creators and collaborators.

Our Operations and Engineering Team is engaged ferociously since midnight Dec 6 to address this issue. We expect a speedy resolution of this issue soon.

Right now we just deployed a banner in-product that will take you into offline mode if you haven't already done so. More details to come, and again apologies for the downtime.

Hey all, 

We are experiencing an outage that is impacting all data access interactions.  These include but not limited to Open, Save, Upload, Export and Viewing. We recommend that you work in offline mode for the time being. We are working actively to correct the problem and will update this thread once it is resolved.  We apologize for the inconvenience and appreciate your patience.

You can also keep track of the status via the health dashboard. 

Here's how to switch to offline mode to continue working while we dig into the issue. 

https://knowledge.autodesk.com/support/fusion-360/troubleshooting/caas/sfdcarticles/sfdcarticles/How...

If you restarted Fusion without going into offline mode before, it may seem like fusion is hanging at the launch screen. Wait a few minutes and you should be able to get in. As soon as you get in, switch to offline mode. Then you should be good to go while we fix the downtime.

[O.Tan]

Part of the reason why it's not implemented yet is mainly cause on the server side, the data isn't stored like a typical file-folder structure. But from what I've read, there's a solution in the works, we just have to wait for it.

[brianrepp]

Hey @benAZ3YM - indeed, it's been a rough day.  I assure you, we're listening and we understand the position this puts everyone in.  This is a long thread so wanted to make sure you saw the post from Prabakar a few hours ago here.  He will be posting more details in the morning (PST) about what we're doing to prevent this kind of a severe impact in the future.

[benAZ3YM]

@O.Tan I'm sure you're right, things are always more complex behind the scenes than they appear (I know, I've built a number of them 🙂 My point is that this should now be prioritised to work in a way which makes sense for the users, not just for Autodesk.

[3Dpcb]

Great that the Fusion Team will do something about the data access...

"Once we get thru this and you are all up and running, I will update you on the work we are doing and are doubling down to make Fusion resilient and more importantly make sure that you have access to your data all the time."

This outage is of course very inconvenient, but for me it is just like a small bump on the road compared to all the benefits, I have experienced by using Fusion 360.

[dr.nsharma]

Developers at Autodesk,

I sincerely wish that your rollback or roll-forward works and that you can get this resolved soon.

Thanks,

Naresh

[brianrepp]

Thanks @dr.nsharma -   thanks for the wishes, things are looking very good right now on our testing.

[daniel_lyall]

it's only 1/4 to 3 you got 4 more hours on your side of the pond,  9 on my side good luck

[i.geukes]

I know this is not helping. However I need to make sure you understand that we now have 3 engineers for almost 1,5 day that are not able to get any work done.
Not to mention the implications for the rest of the organisation. I do not need to tell you what the costs are...
Conclusion: continuous local storage of files is a MUST for a future Fusion update.

[Neil_Cross]

I must admit, I was staggered and a little saddened to see that the service is still offline well into the next working day here in the UK.  Good luck getting it going guys, I appreciate how hard you'll all be working behind the scenes to rectify this.  

[brianrepp · Accepted solution]

All - good news!  We're currently ramping back up services.  There may be some instability during this phase, but would appreciate any feedback if something isn't appearing to come back online correctly.

[scottmoyse]

Well done. 

[scottmoyse]

It looks like some of the Fusion Team Hub side of things is slow at the moment. I'm getting some Service Unavailable message and folder contents aren't loading. You guys should have access to the Autodesk CAM team site, is it happening at your end as well?

[dr.nsharma]

@keqingsong @brianrepp @prabhakar and others at the Autodesk Development team.

Great job guys, its working for me in Italy. Hope that's the case for others as well. 

It must have been a devastating experience for the developers, and my sympathies. That said, the management should pay serious attention to the possibility of this reoccurring and to address a solution that makes customers confident to not drop the ball with such a fantastic product.

Thanks for your efforts,

Naresh

[susibiker]

---

@brianrepp wrote:

All - good news!  We're currently ramping back up services.  There may be some instability during this phase, but would appreciate any feedback if something isn't appearing to come back online correctly.

---

Well...

It all seems to be OK thus far. Uploads are slow, but probably what you would expect with everyone hitting the servers at the same time.

I'd just like to say THANK YOU for the Heroic effort you and your associates have obviously put into fixing this issue. Thank you too for your positive attitude and taking the time out to engage with us and keep us updated during such a trying period. I wish other companies could be as good (we all have our own personal list of companies that fall down in this matter)!

As hard as this issue has been for some folk, it has certainly served to highlight the shortcomings in F360 when the Cloud service fails. As long as we all learn from this, and something positive comes out of it, it will not have been a total loss.

Thank you again.

Susi xx

[scottmoyse]

Also Hub switching isn't working intermittently. I'm guessing it will resolve itself in time?

[brianrepp]

@dr.nsharma@susibiker - thank you guys both for the kind notes, i'll be sure to share with the team.  Thank YOU for sticking with us through the outage, and keep an eye out for our post later in the day talking about next steps.

[brianrepp]

@scottmoyse - we're looking into it now, thanks for reporting!

[S.Kievit]

We are also back online in the Netherlands.

Thank you, to the autodesk teams.

Stefan Kievit

[scottmoyse]

I'm back to being Offline again. I'm sure it will be fine by the time I get up in the morning. Night 🙂

[PierreChapuis]

Fusion 360 is working now for me in France.

Thanks to the engineer team for the fix !

Best.