Initial Vault Migration - Batch Adding Files to Vault

[e_frissell]

Hi guys, I was recently re-hired by a previous employer and one thing they'd like me to help with is getting their vault going.  They've had a vault for roughly 6 or so years but have struggled to get it going for various reasons (mostly a combination of many years of historical data, massive assemblies, and difficulties with support).  To add to the excitement the company does have several satellite facilities that have been working out of the vault who use the same ERP system, therefore we expect to run into the issue where a part will already exist in vault and have the same part number as one of parts in our assemblies such as bolts, washers - the common stuff.  We recognize that we're in a pickle.  If I were to estimate our machines have 10~30,000 instances of parts.  They are massive.  Secondly I'd bet that we have 300,000~500,000 unique part numbers having started with Inventor in the early 2000s, so it's a pretty large pickle we're in.  In 2018 an attempt was made to pack n go an assembly into the vault.  It ran for 2 days and ended when the vault (or Inventor) crashed.

Secondly I don't expect our VAR/support to be worth much.  We had a meeting with them 2 weeks ago where we discussed the above issue and emphasized the size of our assemblies and how we anticipate a lot of difficulty getting things into the vault.  They requested that we give them one of our assemblies and they'll see what they can do.  We gave them a pack and go and when we reached out to see how it went they responded with "No problems, we opened your file just fine!  We didn't try adding your file to the vault because we don't have a vault project file configured for your environment."  In follow up emails they refused to answer any of my specific questions regarding the situations we'll find ourselves in, but they are planning on setting up a meeting with their vault expert so hopefully he'll have better advice, but we're not off to a good start so hopefully this post can help provide us with some general direction, or at least be able to communicate our intentions with our support better.

So with all this in mind I am trying to move this issue forward, and figure out how to plan for many of the situations I expect to encounter.  We're getting the high level details sorted out and at a minimum will add the most recent versions of our core machines to the vault and then transfer historical stuff on an as-needed basis, but if it became feasible to add all our historical information then perhaps we go that route.

If this was your task, any thoughts on how you would proceed? 

Maintaining Links

The duplicate parts is likely going to be our biggest headache so we want to minimize the chance of breaking links.  Would this mean the AutoLoader becomes our best bet?  Let's say we upload Machine 1 that contains Part A, Part B, and Part C.  We begin uploading Machine 2 that contains Part C, Part D, and Part E.  Would this result in an upload error because Part C already exists in the vault?  If yes, do we need to upload Machine 1 and Machine 2 all at the same time?  If no will the AutoLoader skip trying to load Part C and be able to maintain the link to Machine 2 even though they were checked in separately?  Since bolts and washers will be shared on a litany of machines what's the chance we can upload 200,000 parts into the vault at one time via the AutoLoader?

Duplicate Parts

We expect to have an issue with the satellite offices having created several components on their own that we are also using in our machines.  For obvious reasons we need to prevent duplicate parts, or at least handle them in a way that doesn't create more headaches.  Prefix/Suffix is one route and rename would prevent the issue entirely but renaming 100,000 parts just to rename them again seems like wasted time, but it meets the goal of retaining links and we can handle things on a case by case basis later.  Alternatively all of our manufactured parts start with a 4 digit number then a dash (1234-DA) while our purchased parts utilize a 3 digit code - (123-45678) therefore between some clever coding to replace parts, a copy design, etc... it wouldn't be that difficult to remove a suffix.  

Or alternatively could we turn off "force unique part numbers", make the upload, find duplicate part numbers, turn one inactive and rename the least used occurrence to a "-update" to force people to update to the more commonly used part number?  Is there an 'inactive' status to let something exist (i.e. a duplicate part number but different file metadata) but ensure that it's not able to be found/used later?

Any advice or things to look into would be appreciated

[swalton · Accepted solution]

Sounds like Fun!!!

Let me test my understanding:

1. You have 2 or more design offices
2. At least one office is using Vault (Pro or Basic?)
3. You want to merge all the work into a common Vault Database and Filestore.
   1. You do not want to a Vault for Office 1, and a Vault for Office 2 that can not share data.

Assuming the above is correct, I'd look at the following:

1. Set a cut-off date to transition to Vault for each office for new projects.
   1. When you are in a hole, STOP DIGGING. 🙂
   2. Run for awhile with the old projects on the old server and new work in the common Vault.
   3. I understand that some types of work may not be possible to separate this way.
2. Explore using a Staging Vault to import and fix old designs. 
   1. Allow duplicate file names during design uploads.
   2. Rename hardware with prefixes, etc.
   3. Clean up any existing duplicate files too.
   4. Fix ERP numbers, descriptions, other iProperties using Vault's editing tools.
   5. Once you have a clean dataset, Pack-n-go from the Staging Vault and import the files into the Production Vault.
   6. Make a judgement call to see if you want to do the import/cleanup on a per-design basis, or just import all of the old designs into the Staging Vault at once.
3. Once the old designs are in the Production Vault, don't touch them.
4. If you are using Vault Pro with Lifecycles and Revision Control, consider putting the old designs in a "Historical" Lifecycle and State.
5. As you need to make revisions to the old designs, think about using Copy-Design to create new files and replace the hardware and duplicate components with the existing good data from the Production Vault
6. Obsolete the old-design files as updated ones are created.
7. If you can't use Copy-Design, check out the old assemblies and use Inventor or Design Assistant to replace the old sub-components with the correct ones from the Production Vault.

My goal is to break the work up into smaller chunks.  You may be weeks, months or years in the migration process, but the day-to-day work load will be smaller.  It will depend on how often you have to update and modify existing designs.

We found an independent Vault Implementation Consultant that was very helpful, instead of using a Reseller.  There are also some folks who write custom Job Processor tools and ERP integrations, independently from the Reseller network.  PM me if you want the names.  

[Gabriel_Watson · Accepted solution]

#1 - Any attempt to start up a Vault with full capability of over 20 years of data stored elsewhere is impossible. You HAVE to set cut-off dates and limitations of a fresh start, preferrably learning from previous mistakes. See this as an opportunity to forget bad design practices and botched files, a unique chance in the history of your company to utilize the extensive design experience you gathered in those many years to now define a better foundation for future projects and newcomers.

#2 - You have two routes:
a) either shoving every part and assembly you can into Vault to maintain backwards compatibilty (keeping unique file names is impossible here, and folder structures wil need to be aware of differences in project/IPJ configuration, perhaps with more than one Vault project), or:
b) treat Vault as a repository of "pristine" and new information. Only adding new projects or curated historical data (projects and libraries) that will be relevant in the future. Usually, in this case, you keep the previous network drives or other database prior to Vault for your team to access it as needed (they can port stuff into Vault as long as a filter is run on the files, renaming them and adapting to the Vault structure, but preferrably keep all ol data separate).
Personally, I have seen and recommend the second approach, which requires your team to port only libraries and older projects on a needed basis. If it seems like a lot of work, well, it is! But it is still better, in my opinion, than keeping the "multiverse" of old design data into one jungle of differing configurations.

#3 - The autoloader is fine, but my recommendation is to invest a lot of attention in migrating this data in. Making sure all necessary properties are included, and uploading one or a few projects/customer folders at a time. You should not use Autoloader in several machines at a time to upload the content faster, unless you are clearly desperate. I am also a firm believer that the "enforce unique file names" setting is overrated and not as necessary as most people think, and the only way to move into Vault massive amounts of data like that is to ignore duplicated file names at least temporarily (deal with those later using searches and other tools).

#4 - Luckily, and thanks to other beta testers like me, we have now a duplicate search in Vault that is easy to use and can capture geometry duplication, not just by file name. You may need a small subset of 2-3 designers working with you on that to truly filter the whole content, and to educate others in using this tool during design reviews. I agree with renaming by adding a suffix if you have to carry all the content in, but you want to keep this to a minimum and only move design in as needed. Draw a line somewhere. You can use a lifecycle state of "Obsolete" or other as you see files that are duplicated and replaceable. You can tailor this obsoletion to be a soft obsoletion (still allowing users to open the assemblies resolved) or hard one (obsolete files can only be read but not downloaded), which would force people to resolve their assemblies when opening them, and re-linking with new/better content.

#5 - Always look up to your "SSOT" (single source of truth), be that the ERP system or the customer part numbering provided for a project. You need to set up your Vault Numbering Schemes to make sense for your release cycle and change cycle. In any decision, you can always look up to these as a guiding vision of what should be the best outcome and best way to treat the data.

#6 - Totally agree with @swalton above, specially in that you will most likely need a "quarantine" space in Vault if you are fixed on the idea of moving everything in. I just personally prefer (and have seen it work) where you will forever keep the old system in parallel to Vault, and eventually all the old design database simply isn't used anymore (as most relevant data has been ported in via Pack and Go and careful migration of properties/etc.).

Good luck!

[christian_gessner · Accepted solution]

Hi Eric,

Autodesk provides a tool for Vault called Vault DTU that exports and imports Vault data including the binary files, file metadata, file history, folder structure, items and so on: https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Autodesk-Vaults-Da...This tool basically writes out all the data from Vault - except ECOs (Change Orders) and can later import it to another Vault.

This way you can offline clean-up you data and import it to a new Vault database.

Since most of the data that gets exported by this tool is stored in a single (very cryptic) XML file, manipulating this file is almost impossible for Vaults with big amount of files and versions. This is why we offer utilities that can translate huge Vault BCP packages into an (easy to understand) intermediate SQL database, giving you the possibility to rename files, move them to different folders or remove duplicate parts while maintaining and re-routing the links/references to the remaining files. After the clean-up the utilities translate the data back to the XML format that is needed to load the data to a fresh Vault. You can find more information here: https://www.coolorange.com/powerload

The Vault DTU is only available to certified consultants and partners so you cannot just download it. But you can reach out to your reseller and ask for this technology or even PM me directly if this sounds like an interesting approach for you.

Thanks,

Christian