RESOLVED: Fusion 360 Service Outage

[keqingsong]

UPDATE DEC 8, 2;22 A.M. PST -

All, tests on our end have shown that we are back online. You should now be able to log in and access your data and use Fusion 360 as you normally would. Things are still ramping up, but if you continue notice any sluggishness or issues, please let us know. Thank you for your patience, sympathy, words of encouragement, and continued support. It meant a lot to all of us!  

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 9:17 P.M. PST -

All – A quick update on the outage.  We are close to a resolution.  We have started testing Fusion in our internal staging environment.  Once we are sure everything works, we will go live and bring Fusion online. I know this has been very painful for all of you and all of us in the fusion team feel your pain.  I really appreciate your patience as we work the problem.  Once we get thru this and you are all up and running, I will update you on the work we are doing and are doubling down to make Fusion resilient and more importantly make sure that you have access to your data all the time.

In case you are wondering what the current issue is why it is taking so long to address it, here are the details.  Fusion 360 is a network of services and one such service is what we call ACM (Access Control Manager) Service.  The purpose of ACM is to govern project level access to all data resources (items, projects, folders…) and its derivatives (views, comments, …).  We periodically optimize our Fusion Item Access Control Database, based on changing application needs and usage of ACM.  An optimization done on Dec 6 made calls to ACM getting locked up in the database and never return. Ultimately, we had to bring our ACM service down to roll back the changes and make sure all the services operate normally.  It is an involved effort and hence the delay.

Prabakar.

----------------------------------------------------------------------------------------------------------------------------------------------------

UPDATE DEC 7, 11:05 A.M. PST - We are now estimating to have the outage resolved by 1pm PST.  I will keep this thread updated if anything changes.  Thanks for your continued patience.

UPDATE DEC 7, 7:19 A.M. PST - We are making progress on battling the outage. We have been having issues with our Access Control System that governs project level access to all data resources (items, projects, folders...) and its derivatives (views, comments, ...) in Fusion for creators and collaborators.

Our Operations and Engineering Team is engaged ferociously since midnight Dec 6 to address this issue. We expect a speedy resolution of this issue soon.

Right now we just deployed a banner in-product that will take you into offline mode if you haven't already done so. More details to come, and again apologies for the downtime.

Hey all, 

We are experiencing an outage that is impacting all data access interactions.  These include but not limited to Open, Save, Upload, Export and Viewing. We recommend that you work in offline mode for the time being. We are working actively to correct the problem and will update this thread once it is resolved.  We apologize for the inconvenience and appreciate your patience.

You can also keep track of the status via the health dashboard. 

Here's how to switch to offline mode to continue working while we dig into the issue. 

https://knowledge.autodesk.com/support/fusion-360/troubleshooting/caas/sfdcarticles/sfdcarticles/How...

If you restarted Fusion without going into offline mode before, it may seem like fusion is hanging at the launch screen. Wait a few minutes and you should be able to get in. As soon as you get in, switch to offline mode. Then you should be good to go while we fix the downtime.

[SGL-Design]

Do you have a link to something that tells about that?  Is that what "Connected Desktop" is in the Fusion 360 Roadmap seen here?https://app.mural.ly/t/timerahart2/m/timerahart2/1478130625217/view/1994270530?autoplay=true

Thanks

[fulcrumusa]

---

@charegb wrote:

@GMDesigns74 I think I have a work around for this error.

1) Close Fusion

2) Go to %localappdata%\Autodesk in windows explorer

3) Rename the "Web Services" folder to something like "Web Services backup"

4) Launch Fusion - It will prompt you to login and then show you all the cached data in offline mode.

For any Mac users reading this, the location is - /Users/$username/Library/Application Support/Autodesk

Let us know if this works.

Thanks,

Bankim

---

Bankim,

This does not work. I still get the same error message.

[neljoshua]

This is anecdotal evidence, but before I go home I wanted to mention something.

For the past week, I have not been able to get through a day without a bunch (somewhere in the 4-10 range) of Fusion crashes.  Today while working offline, Fusion did not crash once (while working with the same assembly that I have been).  So, I guess there is a bit of a silver lining, at least for me.

I am thankful I did not have any deadlines & hoping that those of you who do are able to mitigate the issues caused by the outage.

[rodwyll]

@SGL-Design Yes, The project was originally called connected desktop, but now it is part of a360 and is called a360 Drive or autodesk Drive ( they really like changing names)

https://a360.autodesk.com/drive/   at least thats how I understand it. last I checked it was in beta. dont know if its published now, but since the servers are down I cant verify that. 

http://forums.autodesk.com/t5/fusion-lifecycle-forum/fusion-360-fusion-lifecycle-connected-desktop-a...

[SGL-Design]

Thanks, I'll have to look into that a bit more. 

[scottmoyse]

A360 Drive was Autodesks first attempt at cloud storage and desktop syncing. It's installed with most of their desktop software. It's crap. Connected Desktop hasn't been delivered yet, since that will hook into the A360 Hub version of Autodesk 360, not the disjointed Drive product.

[cekuhnen]

@brianrepp @keqingsong

Ops clicked accept as solution by accident.

today's example should be a good reminder that the current file management has serious problems when the server crashes - which let's be honest also happens to Apple Google and co.

I personally like the cloud part a lot however it is seriously imperative that users can pin projects for local storage and keep them there. As long as they need.

Let the user decide how much storage they want to dedicate to it. Ever cloud data sync app allows you to sync everything or only partial documents/folders.

this would be fantastic if implemented in fusion.

[Anonymous]

seriously  how many times are you planning to **** the  ETA just bloody remoe it its embarrecing now. and i am never using fusion ever again 

[cekuhnen]

@Anonymous Come on buddy - yelling does help. They thought they could fix it but it seemed as things were not so easy to solve.

do you prefer them not saying anything? I canceled two classes today. Stuff sometimes breaks - annoying but reality.

[susibiker]

---

@neljoshua wrote:

For the past week, I have not been able to get through a day without a bunch (somewhere in the 4-10 range) of Fusion crashes.  Today while working offline, Fusion did not crash once (while working with the same assembly that I have been).  So, I guess there is a bit of a silver lining, at least for me.

---

Ah. Thanks for this comment!

I am new to Fusion 360 - only been using it for a couple of weeks. I thought all these crashes was due to something I was doing. In the end, every time I do a significant edit, I manually save the drawing, then export every other one to my local drive.

I also concur - since the outage, no crashes. It sometimes vanishes off somewhere doing who knows what, but it comes back again. Sounds like a lot of these crashes are due to the cloud autosave 'feature.' Or are at least possibly related.

Susi xx

My PC should be OK to run this, right...?

Core i7-4390K 3.4GHz (6/12 cores). 64GB RAM (Corsair Mental-DDR-something-or-other...). GeForce GTX770. Win7x64Ult. 1TB Samsung SSD 840 EVO

When Fusion 360 hangs, Task Tanager often shows Fusion 360 using about 900MB of RAM and using about 4%-6% of total PC processing power, and the video card is pretty much asleep too.

Methinks that no matter how good this program is when it's working, it is not ready for Prime Time yet. Doubly so with the over-reliance on 'Cloud' <spit!>

And that's just sad: This is the first time in years I've come across a program that has real potential to open up the possibilities of CAD to everyone.

[Anonymous]

the only thing worse than a buggy platform is false promises. i have been up  from 28hrs just waiting to get acsess to my files back this is rediculous. 

[fulcrumusa]

---

@Anonymous wrote:

---

@neljoshua wrote:

For the past week, I have not been able to get through a day without a bunch (somewhere in the 4-10 range) of Fusion crashes.  Today while working offline, Fusion did not crash once (while working with the same assembly that I have been).  So, I guess there is a bit of a silver lining, at least for me.

---

Ah. Thanks for this comment!

I am new to Fusion 360 - only been using it for a couple of weeks. I thought all these crashes was due to something I was doing. In the end, every time I do a significant edit, I manually save the drawing, then export every other one to my local drive.

I also concur - since the outage, no crashes. It sometimes vanishes off somewhere doing who knows what, but it comes back again. Sounds like a lot of these crashes are to to the cloud autosave 'feature.' Or are at least possibly related.

It only seems logical to conclude that and that should be a pretty big clue to the Autodesk devs when debugging the crashes.

When Fusion 360 hangs, Task Tanager often shows Fusion 360 using about 900MB of RAM and using about 4%-6% of total PC processing power, and the video card is pretty much asleep too.

Methinks that no matter how good this program is when it's working, it is not ready for Prime Time yet. Doubly so with the over-reliance on 'Cloud' <spit!>

And that's just sad: This is the first time in years I've come across a program that has real potential to open up the possibilities of CAD to everyone.

---

It is entirely possible that Fusion is blocking on network access to the cloud. That's why you won't see much resources being used but the program is stuck. That is also why, blocking operations are usually done in a background thread when there is a UI involved.

[scottmoyse]

---

@cekuhnen wrote:

Ops clicked accept as solution by accident.

---

I've removed it as the solution so people don't see this in the forum as solved and get their hopes up.

[scottmoyse]

Looks like we hit the 6pm marker... or 2 minutes from it, and now we are looking at being delayed another 6 hours:

[scottmoyse]

Looks like we hit the 6pm marker... or 2 minutes from it, and now we are looking at being delayed another 6 hours:

[susibiker]

Thanks fulcrumusa.

I've had so much fun learning F360. I *REALLY* like it. I want it to work.

I'm not angry at all, but I truly hate this 'Cloud' sh** with a passion.

I feel for the poor souls at AutoDesk trying to sort this out. Even more, if it turns out that it was a mistake made by some poor overworked drone who did something by accident.

Let's face it - we've all been there.

I had one today. When F360 went down, I struggled to get my current design finished. All done, then I noticed that I had somehow managed to get some belt idlers totally at the wrong height in the middle of a complex (for an F360 novice -> me) 3D build. I noticed four hours ago - just after I sent the files to the fabricator!

Anyway, I've been at this for nearly 22 hours straight. Honestly.

I'm going to call it a night (it's 0200 here).

Good luck to you all, fellow sufferers and AutoDesk employees alike.

Susi xx

[cekuhnen]

Uplay is also down - seems something else is rather going in the net in general.

[kylek292]

Hopefully something good comes out of this, sometimes it takes a massive outage for management to finally allocate proper resources or give the green-light on the offline features.

Because I was on vacation over the holidays and hadn't launched 360 in 2 weeks time, I'm one of the ones stuck without even an offline mode. I can model fine, but I can't save, at all.

[mslatem]

Relax Max! - Getting angry at a situation you have ZERO control over will just make you more frustrated. I am sure they are working as hard as possible to remedy this problem and must be under enormous pressure. Once the dust has settled, I would be shocked if they do not focus on a permanent solution to prevent this happening in the future.

My 2 cents:

Put an immediate feature freeze on all the new "bells as whistles" features that are planned. Dedicate all engineering resources to implement a no-brainer local storage architecture for Fusion360 that has no underlying reliance on the Cloud to function. This should happen as a matter of urgency & be the #1 priority.

Reliability, Robustness and Cloud independence is an absolute priority! 

[CarlM89]

---

@mslatem wrote:

Once the dust has settled, I would be shocked if they do not focus on a permanent solution to prevent this happening in the future.

---

I'd consider my wasted time well worth it if this event caused Autodesk to rethink their cloud strategy and implement something that doesn't shut down their entire customer base every time their server has a hiccough. They have proven to be incredibly responsive to customer requests and suggestions except in this area. Here's hoping!