Project file fails - Reload families, "random" corruption

[Anonymous]

on 1/24 one of our files started having problems so that no one could sync to central.  Due to a deadline we resolved by creating a new central but the supposed problem was caused by corrupt families.  We removed the 4 corrupted families and moved on.  They weren't critical to the project but we weren't clear as to why these families were corrupt, we were directed to this article:

https://knowledge.autodesk.com/support/revit-products/troubleshooting/caas/sfdcarticles/sfdcarticles...

Cosmic rays? Phase of the moon? who knows.  Random.  

Friday (2/3) we started getting errors that users could not edit family parameters. Type or instance, nothing in a family could be adjusted without an error that the family needed to be reloaded.  Initially we thought it was isolated to one or two families, further digging indicated it was any family we touched.  Reverting to previous versions did not help, even going back a day.  After 4 hours of troubleshooting I contact support again.  This is the supposed fix:

https://knowledge.autodesk.com/support/revit-products/troubleshooting/caas/sfdcarticles/sfdcarticles...

That article indicates reloading one or two "bad" families.  In our case it was 487.  From what I can tell there are about 10 that AREN'T corrupt.  When I called autodesk, the solution was to look at journal from latest file, collect a list of all corrupt families.  Then find the last known good file, export all families as a source for reload.  I've done this, it generates about 400 errors of dimensions being removed and constraints being destroyed.  Along with a few complex family types (nested shared families) being deleted, despite being reloaded from sources.  In finding the latest "good" file, I realized that this corruption, of over 400 families, happened within 5 hours from 2/2 at 8am to 2/2 at 1pm.  But Autodesk swears it is random?  

The original issue is now being reported by others through our local user group.  And I think that burying critical info away in the journal during an audit a pretty poor way to notify users of a potential disaster brewing.   

[brendan_mullins]

Tom,

  I really appreciate you taking the time to give a prompt and thorough response.  It was interesting to find this thread because it was identical to our issue, except we are sadly in CD's.

  We are going to upgrade to the latest Revit 2016 build at the same time we try to repair the file in the hopes a patch for this hidden problem was made in 2016.1.8.

  Our file seems to have every single family corrupted, which from what I am gathering is what happened to you as well.

  My real question is when you did all this during SD's, did your file survive all the way through the end of the project?  Also, how did you not decide to just throw the work out and go to a backup?  Lastly, was your model on C4R or Revit Server? 

  Thanks again for the help.  Any and all advice is very useful as we try to save this dying model without reverting back 7 days of CD work.

-Brendan Mullins

[coffeyj]

Hi Brendan,

I would also be interested to hear if this was a consideration for Tom, what course of action he chose and why, and what the results he observed were.

That said, if your choice is between reloading all the families and rolling back to a previous version of the file, I would highly recommend that you get the current 2016 update and reload all of the families. I say this because it will be difficult for you to determine if the backup version of the file you select is free of this type of corruption. You may pick a backup copy that only has a few bad families, so at first the file would be usable for a while, but ultimately you would yourself in this position again in (likely) the very near future. Reloading the families can be tedious and time consuming, but it is the best way to rid your file of this type of corruption.

I hope this helps. Please let me know if you experience more trouble after updating 2016, opening with audit and reloading the problematic families.

Best,

Jill 

[Anonymous]

Jill,

We verified a past version was viable as source of families by running audits on all previous versions until it was clear of the error.  From there we exported all families out.  Took me about 8 hours of work to repair the file.  It is worth noting we didn't find a version where only a few families were corrupt.  It was either all or nothing.  

Because of the client and our timeline we didn't have much of an option.  Loose a few dozen manhours of work (rolling back 4 days worth of work) or me spending the time to hopefully repair it were our options.  Our biggest loss was all the elevations being damaged, a day lost where no work could be done, and the tedious work of repairing the file. 

Once we reloaded everything we deleted anything we couldn't repair (I think I mentioned the elevations in a previous post, that was the most damaging).  After the looming deadline I did some maintenance on the file.  I audited it again, purged judiciously, re-authored file.  At that point I put the file back into production and let them run with it.  

Because of concerns with everyone being on the same version and the difficulty of upgrading an entire office to the latest we may not be at the most current version.  But the project is nearly done and since this incident hasn't had a recurrence of this particular problem.  One thing we have tried to make people aware of is leaving files open for more than a day.  Due to its size, this project took quite a while to load.  So some users just left the model open overnight.  From some reports from Autodesk I've heard that could be part of the problem.  

Is that what you were looking for Jill?

[Anonymous]

Someone has split this off into this forum.  The previous replies are lost here: 

https://forums.autodesk.com/t5/revit-architecture-forum/project-file-fails-reload-families-quot-rand...

@Viveka_CD This seems a bit damaging as others won't see all the responses and from the previous thread it now appears both me and @coffeyj are talking to no one.  

[Viveka_CD]

Hi @Anonymous

The posts have been merged with the original thread as requested. It was split into a new post because of a C4R issue by Brendan.

@brendan_mullins please create a new post in the Revit collaboration forum since your issue is related to C4R

Regards,

[brendan_mullins]

Yes, that forum split was strange since we are having/had the exact same issue.

I am updated to the most current Revit build and am now swapping families with some from 2 weeks ago.

I will then audit again and search for the straggling corruption.

From what I am gathering from our Autodesk Support Team (we are privileged with enterprise help) this is indeed the method to solve this.

I am also learning that another massive C4R job within my firm suffered the same fate and I am going to learn how they survived.  From the limited info I learned it was by doing what Tom and Autodesk have recommended : Updating Revit, Auditing, Replacing with Old Families, Cleaning and Searching for other corruption.

Apparently it is survivable, but up to know it has not felt like that was possible.

I will keep this forum posted with how this goes.

[Viveka_CD]

Hi @brendan_mullins

The protocol is to move or split posts to respective forums based on whether an issue is on C4R, local or on a Revit server. We have a separate forum for C4R issues.

I wanted to clarify that the move/ necros thread split was based on what you wrote earlier:

___________________________

We are experiencing this exact same issue right now in the exact same build of Revit.  However, the only difference is we are on C4R.  Was this solved?  We are working to upgrade to the more recent build, but any advice is appreciated.  This is taking our entire team down right before a CD deadline.  Let me know, thank you.

________________________

Also, could you please confirm your case number? This will help avoid duplication since an assigned specialist is already working on the case.

Regards,

[brendan_mullins]

Though I understand why you split it, I now have this thread, my own Autodesk contact, and another project within my firm with their own Autodesk contact, and no one inside or outside of Autodesk knows why this happens.  This is the only forum post I can find anywhere on the internet with this issue (though surely others have this problem), and from what it seems like Autodesk does not know if this is a C4R issue, Revit Server issue, or Revit version issue.  I am hoping Autodesk can find the cause to save others the financial loss people are experiencing from this near fatal issue.

I have upgraded my entire team to Revit 2016.1.8, audited and/or reloaded all families from a previous file, and have now let my entire team (12 users) back into the model.  They have successfully been working and syncing for a whole day.

The other project within my firm that had the same issue never upgraded to 2016.1.8 (they stayed in Update7 R2), and they ended up having to reload all families 10 times over the course of 6 months before the problem went away.  This resulted in serious financial loss due to the waist of billable time, and need to hire someone full-time to simply clean and manage families while this occurred.  I am hoping the versioning was the problem and we don't suffer the same fate.  At this point, that is the only difference between them and us.

I will update this post in a few days if we sail smoothly through the week.

Thanks again everyone for the help.  If this is indeed a Update 7 R2 issue, Autodesk should notify all folks using this version of the bug.  This can easily cost projects tens of thousands of dollars.  However, like I said, I have no evidence yet that it was that build or something else.  I am hoping my Autodesk Rep finds the root cause.

I will provide a case number as soon as my rep responds.

-Brendan

This seems to be the Autodesk post to solve this problem.  I just hope to determine a cause.

https://knowledge.autodesk.com/support/revit-products/troubleshooting/caas/sfdcarticles/sfdcarticles...

[brendan_mullins]

We have now been working successfully for a week and audits are not reporting any more corruption.  Unless we go down again, the problem seems solved.  Wish I had a cause, but so far still no word from Autodesk.  Thanks again for the responses.  Fingers crossed it's smooth sailing through the rest of CD's.  We are more seriously considering upgrading to Revit 2018 and likely will at some point.

Take care, all.

[BIMologist_]

@brendan_mullins, @Anonymous 

It is so frustrating when you have so many people sitting twiddling there thumbs and all the pressure is on you to fix the problem and fast, without loosing work. It is tough and I have been in your shoes many times in our office. The biggest one has been the "Too many Elements Missing". It is from that frustration and learning from I had enough material to present at AU. 

I have been dealing with these issues as well and wanted to share the presentation and documentation and some items, that do not FIX your issue, but may help diagnosing them. @Anonymous and I presented it:

BLD125158 - Code Blue Dr Revit - How to Resuscitate Corrupt Revit Models Presentation
http://bit.ly/2BQ38qe Watch it at au.autodesk.com

Handout from my presentation and Datasets
https://1drv.ms/f/s!AmO3IjYenWEdhPYvoPS5MKqY9aSjeQ

I hope this sheds some light.

[ic349]

This has become a weekly issue for us over the last two months.  It rarely happened on network drives, happened quite a bit on Revit server projects, and now that we have migrated all new inter-office projects to C4R we are repairing files EVERY SINGLE WEEK.  We are running Revit 2018.2.  The 2016 projects have less incidents in C4R than 2018.  Some things to note after reviewing this thread:

• We only roll out even year versions of Revit for production (...,2014, 2016,2018) We got sick of the bleeding edge issues with odd year releases.
• Our library is on a network drive and is updated every other year as well.  We keep Revit pointed to their respective library years.
• corruption issue seems to spread like a cancer, and have recurred on files that were rebuilt from SCRATCH
• families that are added to a project that are not in our library (BIMObject, etc) are to be saved to a networked location, audited and reviewed and loaded from the network location and NOT from memory.  This gives us a recover point.

Most recently I have chatted with Nauman and followed the advice to save out the families in bulk from an open project.  Revit will fail to write any with corruption.  This revealed that there were not in fact hundreds of corrupt families but only one or two.  The relationship has turned out to be shared materials and assets: thus the cancer like action of spreading. 

• if you are working in C4R look for materials that will purge with a GUID and not a real name, families too..
• open with audit often fails to load the file and Revit crashes - open normally and do a bulk family save in this case
  • replace any corrupted families that will not save, purge...purge...purge.
  • save with COMPACT CENTRAL MODEL option

We are trying to figure out at this point which corrupts first: family or material.  Every single instance we have seen in C4R involves the materials library.  You can literally take every scenario that has been posted in this thread and substitute our names for theirs. 

The end user does not read journals, but that is my group's job.  And we see a lot.  This most recent corruption started with a lot of BIG_GAPs regardless of site location and connection method ( we have eight offices working on these files in C4R at times).  This most recent issue timed perfectly with the outage of the Amazon web services an the slight "blip" of a notification that occurred at health.autodesk.com. 

As with any group of individuals, we have found questionable workflows.  We worked with the teams to minimize and eliminate those.  We have even worked with those teams openly as to what happens on the back end even covering journals as a team.  It is a powerful motivator to them to a) not be the person that did something silly that cost the team time and b) understand how the product works so they can avoid common pitfalls.  For example: don't be the person that is connected to the LAN, shut your laptop lid with the model open and then work at home on WIFI.  Revit DOES get confused where the heck the model is.  And, that is clearly evident in the journal when they do stuff like that. 

Regardless, this is NOT completely random.  They are known issues with common catalysts and they need to be addressed.  I am very interested in knowing how Autodesk is capturing this information and resolving issues on their end.  Us customers are not flippant to preserving our manpower and time and quickly identify and squash poor practices. 

I have been an advocate for a journal library of terms, verbose reporting and less obfuscation in the terminology.  It is cute when the programmers put personal messages in there and if they can do that they can clearly call out what is happening.  We should at least get better tools to recover and combat it.  Unfortunately we are getting very good at shutting down a project and recovering data.  This last shutdown cost us 288 production hours in a 32 hour period.

[brendan_mullins]

I really appreciate you and Nauman taking the time to post more info in this thread.  I had only just seen it.

We are planning a massive upgrade to 2018 this week, so I am saddened to hear the issues can still happen there.  I was hoping the further I got away from the build of 2016 the further I was from this problem.

I think this in one of the most important threads on all of the Autodesk Community due to the shear amount of wasted money that can be caused by this corruption.  No other Revit issue  I have seen can cost an entire team, regardless of project phase, a week of work.  This is very important stuff.

I hope that Autodesk keeps this thread open for others to see and discuss the problem as I am always going to be willing to help them.  I wish this pain and stress on no one.  The worst part was that the fix lands on one person to solve, thus the BIM Manager is sure to have some sleepless nights.  After I talked around the firm, we found another massive job with the same problems.  Exporting and reloading families in combination with purging and auditing was the only fix.  It would be great to have a dynamo script that opens and audits every family after export, but I haven't looked into it.

Autodesk, please help us know exactly what causes this.  The two jobs in Stantec I know this happened to are two of our largest and most important.  We need to know how to avoid it.

Thanks, all.

[ic349]

@brendan_mullins

Well, not a dynamo script that audits as it loads, but this one will load all of the recovered families from a directory.  

https://forum.dynamobim.com/t/load-multiple-families-from-a-folder/11906

It will replace the corrupted families in a project.  There are a couple of tweaks to do to it.  I changed a little snippet in the python code to reload the family + additional type where someone may have purged them.  It does not change the instance values of the family.