Modernize the Vault Upgrade Process

[ThomasRambach]

When you're using Vault in a replicated environment with a large database (1TB+) it can be a very lengthy process to upgrade Vault servers to the latest release. Without doing a risky in-place upgrade to your production Vaults, the process in short is as follows:

• Create a new set of Vault servers with latest version of Vault
• Take production Vaults offline
• Backup Vault data to new servers.
• Restore data.
• Re-replicate data to new subscribers.

This process can take several days with lots of downtime for engineering.

I would like for Autodesk to put some effort into a real time Vault update that would look something like:

• Create new Vault severs with latest version of Vault
• Add new future publisher server as a subscriber of production Vault.
• New servers receive replicated data and migrate to current release and replicate to all new subscribers.
• Users are switched over to new servers and old servers are taken offline.

This ideally would result in zero downtime. Time is money and the upgrade process takes lots of time.

[DanSingleton]

Wow. I thought we were getting long at ~12ish hours for a full backup. 32 is redonk. Eventually, you're just not going to be able to do a full backup because they'll overlap production hours. That's got to be concerning. Anyone figured out a method to do a "live" backup yet?

[steveh3]

@Anonymous....

Yes...we are watching this.

Based on the history of our system....about 11.8 hours per Tb.

So...I can almost double our size which took us 15 years to get where we are at.

Bigger concern is the upgrade process.....30+ hours to backup, 30+ hours to restore on new VM server.

We already have to upgrade on a 3 day weekend...getting close to being forced to only upgrade on four day weekends which doesn't happen often. 

We are the exception...but still has to be on radar.

[david_coulam]

The idea in the original post of essentially using replication to populate the new server is interesting, because most of the data can be copied/loaded, then updated frequently until ready to migrate.  Migration would just require the replication be finished, and access to the old server would need to be stopped.  Then, when migration is completed, if it became a fully independent ADMS, that would be cool.  Server name would still be an issue for replicated environments, but if there were a migration tool that coordinated or watched for upgraded versions of each dependent server, it seems like it could be accomplished.

[gilsdorf_e]

For time reasons we have changed the update process from backup -> restore to new servers -> upgrade new servers to the direct update of the productive environment 2 years ago. Our database is 600GB+ and the filestore 5TB.
Of course, for disaster recovery, you cannot use the standard backup mechanism that vault provides anymore, if you do not want to have 2 days offtime, but you have to use snapshot technology for filestores and databases.

[Boorda]

We are in the same exact boat with millions of files and 30+ hours to back up (without validation), 12+ hours to copy filestores, 24+ to restore. In fact, we're currently in a project where we are scanning in 60 years of historic drawings and documents originally on microfiche and aperture cards and importing them into Vault. We estimate about 15TB of additional file store consumption and roughly 2 million more PDFs. Of course we've decided to split this into multiple physical Vault servers, but this would be highly valuable and probably needed update for Vault upgrades to be viable.

Since the file store seems to be the biggest part of the upgrade, would it be, or is there already a way to host the file store on a completely different server so that all we are essentially doing is upgrading ADMS and the database, leaving the files intack to be migrated by the CAD software if/when used? Something similar to the way the Vault File Server works, but for the Master Vault Server...

[Anonymous]

It is right, but please, operate to simplify the job to end users not to Superusers or administrators first. Vault 's high number of inputs required operations negatively impact on firm costs.

[jan.divis]

Our situation is similar, the full backup takes about 30 hours at that time and datase grows. The upgrade is no longer possible during the standard weekend and causes a design and production downtime. Any shortening the upgrade process is welcome.

[steveh3]

So...here's where we are at today.

153 GB database

3.61 TB filestore

We backup up to a local drive on the same server as adms and filestore. We found that running a backup to a location other than the same server took 30+ hours. Guessing it would 40+ hours now.

What is concerning to me now......

When we first started doing backup locally, it was taking less than 8 hours (filestore was around 3.1 TB). Now, we are approaching 21 hours which is a major concern as our filestore continues to grow. Only about 0.5 TB more and taking twice as long. Doesn't make much sense to me. Nothing else on our system has changed other than Vault updates and Windows Server updates.

We then copy the backup to a Synology device. When we first started it was copy the backup in 12 hours. Now, it's taking 28 hours. Another concern.

For what it's worth, I plan on using Robocopy to implement our next upgrade. I perform a backup and then Robocopy the differentials in the filestore to a new server (keep in mind I am using virtual servers). I have performed this workflow about 10 times now to prove out my procedures with no problems so far. My plan is to white paper this at some point.

Keep in mind I keep the original server intact until the new server is up and running. My estimates are I can have the new server up and running in about 6 hours after the backup is complete. That includes a rebuild of search indexes. If new server implementation doesn't go as expected, then we just continue to use the original server.

Once we get VP2022 going. I'm contemplating on using Robocopy to copy the backup from our server to our Synology device. Based on my tests thus far. I can synch the filestores in abou an hour. Just have to figure out how to get the directories pathing right because the name of the backup folder has a date/time stamp in it.

Stay tuned...

Best,

Steve H.

[steveh3]

Also...for what is worth. About a year ago we spent about $15k for two Synology devices (with redundancies). At any given time I have 4 full backups plus incrementals (I know...over-kill, but I like sleeping at night 😁). Eventually, we will out grow these devices and then I will only have two full backups, but that's a year or so down the road.

 Here's our Synology devices...

[ihayesjr]

With the enhancements to the backup and restore features in Vault 2024, upgrading to a new release of Vault has been greatly improved.