Modernize the Vault Upgrade Process

[ThomasRambach]

When you're using Vault in a replicated environment with a large database (1TB+) it can be a very lengthy process to upgrade Vault servers to the latest release. Without doing a risky in-place upgrade to your production Vaults, the process in short is as follows:

• Create a new set of Vault servers with latest version of Vault
• Take production Vaults offline
• Backup Vault data to new servers.
• Restore data.
• Re-replicate data to new subscribers.

This process can take several days with lots of downtime for engineering.

I would like for Autodesk to put some effort into a real time Vault update that would look something like:

• Create new Vault severs with latest version of Vault
• Add new future publisher server as a subscriber of production Vault.
• New servers receive replicated data and migrate to current release and replicate to all new subscribers.
• Users are switched over to new servers and old servers are taken offline.

This ideally would result in zero downtime. Time is money and the upgrade process takes lots of time.

[ThomasRambach]

When we upgrade Vault from one release to the next, we find it "safer" to create new VM instances to restore our production data to. It allows for testing and configuration to be done offline before they are used in production. The problem is it takes us 50+ hours to backup and validate our Vaults, +12 hours to restore, and another +24 hours to replicate the data back to the new subscribers. This is a lot of downtime during "upgrade weekend".

It would be ideal if Autodesk would put some next generation thinking into how Vault is upgraded and/or moved to new servers. I would love to see the ability where you can transfer the production data from one set of servers on the current release of Vault to a new set of servers with the upgraded version and when everything is copied and synchronized, switch over to the new set of servers without any downtime.

Basically a backup/restore/migrate/upgrade all in one operation and real time to stay in sync with the production vaults. 

This is always the biggest PITA every time we upgrade. In place upgrades are just too risky without any backup plan if something doesn't function correctly.

This will be by pet peeve to annoy Autodeskers at AUT2018 with.

[ihayesjr]

Thank you for posting your idea, we will review this idea for a future release.

[steveh3]

Agree 100%.

Take us 3+ days to upgrade. We can only upgrade over a holiday weekend.

We have two virtual servers side by side (original server and one we intend to go to) in case something goes wrong.

We backup current production server which takes 31 hours, restore to new server which takes 36 hours, filestore compilation takes good day till new server is running efficiently.

All access is off during this upgrade.

If new server fails...still have original server to fall back on.

Not sure what we will do once we get beyond 4 days we grow.

Best,

Steve H.

106+ Gb db

2.5 Tb filestore

[ihayesjr]

At some point in time, the users have to stop using the old server while the new server is being brought up to date from the synchronization step, let say when the process hit 90% synchronized as an example. If not, it will never be completely up to date if they continue to use the old system for them to switch over. Thoughts?

[steveh3]

Irvin...not sure what you are asking here?

But, one thing I have been pondering...

If I cut everybody's access, then do an incremental....

Is it possible to restore a prior backup (2017 or 2018) from the week before to the 2019 ADMS, then before I migrate, restore the four incrementals that I did after that?

I know you can restored incrementals on the same version of ADMS, but not sure about on an upgrade. Possible?

That would save roughly about 25 hours (guessing on time).

Haven't tried this and wondering if it would work?

Best,

Steve H.

[ihayesjr]

You wouldn't be able to restore the incremental on from 2017 or 2018 onto 2019 because 2019 would have already migrated the data and the incremental restore option wouldn't be enabled. However, I think I understand where you are going with that thought. We could try and change the way that works in this scenario.

[ThomasRambach]

@ihayesjr if Autodesk figures this one out, I'll be delighted to say the least. 

[ThomasRambach]

@ihayesjr shortening the cycle from days to hours is the goal.

[scottwayand]

Upgrades that take days to complete means a slower adoption rate for new versions of software.  I'd like to see enhancements made to the software to make it simpler to upgrade.

[uffe-bak]

If you are going to reconsider this area - please also consider the following scenarios:

1: Allow for Subscribers to be off-line during process - and catch up later (We had a power outage on a subscriber killing an Upgrade - with no option for Rollback).

2: Check if a Restore from Production to a Test server, can have an option to let Subscription setup "survive" - so it could be re-enabled on Test to the existing subscribers - in case the in-line Upgrade fails on the Production server.

Thanks,

Uffe Bak

[steveh3]

Agree with ThomasR...

Last time we ended up spinning up a new VM server and tested several restores to make sure that all was good and functionality of upgrade didn't have any performance/functionality issues. Could do this during the week. Then just waited for a 3 day weekend and did the backup from old server and restored to new server.

Kept original server as is until we knew that the upgrade went as expected.

As for the validation of backup...I feel that's just wasted time. That's another 25+ hours I don't have.

If I have two servers...the restore doesn't work, I just go back to old server.

One thing I had mentioned earlier..would be nice to somehow incorporate the incrementals into the upgrade process. Could do the full backup from prior weekend (30+ hours) and then just restore the incrementals (usually only 2+ hours). Would drastically cut down on process.

Just typing outloud 🙂

[DanSingleton]

Just went through the upgrade from Vault Pro 2016 to 2019 two weeks ago. I'd put off the upgrade for some time because I knew it would be an extremely time-consuming process. As it is, it took the better part of two days over a weekend without file access, followed by a couple of days of designer specific configuration afterwards. We're a 7 day (almost 24 hours/day) manufacturing plant so that was not a great thing.  The Vault backup/restore times really need to be addressed and sped up in any way possible.

[Anonymous]

I agree with all the above, whatever Autodesk can do to reduce the time it takes to upgrade.  I'm prepping for a 2016 to 2019 upgrade in the next few weeks and am not looking forward to it.  I recently had to just detach and reattach vaults (to fix a problem breaking the upgrade) and the search indexes are still rebuilding almost a week later.

David P.

[DanSingleton]

Another thing that would help would, at the very least, to be provided with real world default/optional "Autodesk suggested" scripts/tasks for setting up any daily and weekly Vault DB administration tasks and backups as well as  recommended times to run them. Typical CAD Admins aren't going to be full-on DB Admins so it can be a bit daunting trying to set these up the first time. The Autodesk help files for these mission critical administration and backup tasks (and for any Autodesk product in general these days) really leave a lot to be desired.

[ThomasRambach]

@steveh3 I don't know why, but I never have quite enough faith in incremental backups like I do in full backups. 

[steveh3]

@Anonymous..Now you have me scared....

I've only restored incrementals a hand full of times...but I have never had a problem with them. I test my full backup restore quarterly just to make sure it works. Maybe should start doing the incrementals too.

Best,

Steve H.

[ThomasRambach]

@steveh3 

"So, although it saves time taking the backups on a daily basis, it can be more complicated, time consuming and labour intensive to restore Incremental backup, than taking a full backup every day."

https://knowledge.autodesk.com/support/vault-products/troubleshooting/caas/sfdcarticles/sfdcarticles...

[steveh3]

Agree...but in our case. Takes 32 hours to perform a complete backup, and that's without a validation. So, not really an option. We have to use incrementals.

Good conversation and insight though 🙂

[ThomasRambach]

@steveh3 Same here. So the other real issue is backups take too long. I think the backup technology wasn't designed for companies with millions of files in their Vaults.