Vault Migration to New Servers - Full Backup & Incremental

[itsam]

Setup:

We have two servers supporting Vault - A File Store server and a Database server.  We are using Vault 2018.

The plan to migrate Vault to the new servers is:

1. Stop users from checking files in/out of Vault during the migration period (Friday - Sunday)

2. Perform Full Backup of Vault files and databases via ADMS Console 2018.

3. Copy backup to new server.

4. Perform Full Restore of Vault files and databases via ADMS Console 2018.

5. Allow users into new Vault (Monday)

The issue that we have run into is that the backup has taken longer than expected based on previous times.  We did not want to have users down (unable to work), so we've allowed them to access the old Vault on Monday.

Question:  Can we perform an Incremental Backup on the old Vault servers next Friday and then perform an Incremental Restore on the new Vault servers to complete the migration via the ADMS Console?  This should, in theory, reduce the time required.

Thanks,

Thomas

[milindnalgirkar]

Hi Thomas,

How big is your file store and Vault DB? How long does it take for you to copy the backup to the new servers? Presume you are using Vault Pro since you have a separate SQL server. You should not need incremental/delta backup. Run the Vault backup from ADMS command line -- if you do that you can eliminate the step needed to validate the files which is the most time consuming part in the backup process.

I cannot believe there is a non replicated environment (not even that), that cannot be backed up and restored over the weekend. Let me know the filestore, DB sizes and data copy speeds you are dealing with and there should be a workflow where you can shutoff your users on Friday -- and restart on the new servers Monday AM. Provided you are ready to support the process on the weekend 🙂

Thanks,

--Milind

[ihayesjr]

A correction to @milindnalgirkar statement.

A restore, regardless of it being full or incremental, will validate the file store. The server has to make sure that the restore has all of the files the database has a record for. You cannot get around this.

[milindnalgirkar]

Irvin -- thank you for the correction, I had provided half the answer. In my previous experience working with a very large Vault the backup took about 8 hrs from the console and 1-2 hours from the command line. So I meant to say that they will be able to shave time off their backup process.

[itsam]

Thank you for the response.

How big is your File Store?  It is 464.15 GB

How long does it take to copy the backup to the new servers?  About 8.5 hours.

Run the backup from a command line.  I planned to do this, but Autodesk recommended that I did not; but run from ADMS console.  Below is the command that I would have used.  Is this correct?

CD "C:\Program Files\Autodesk\ADMS Professional 2018\ADMS Console"

Connectivity.ADMSConsole.exe -Obackup -BE:\Backups\Full\20200623 -VUAdministrator -VPadminPasswd -DBUsa -DBPsaPasswd -LE:\Backups\Full\20200623\BackupLog.txt

Also, after the restore completed (showing successful), checking revealed some issues (missing files).  I've included some images showing the comparisons between the old server and the new servers using the client Vault application, the ADMS Console, and File Explorer Properties.  I have verified that the backup folders on both the old server and the new server are the same size, so there wasn't and error or failure in copying the backup folder to the new server.  Why is this?  (I'll reply again to upload all of the screenshots).

[itsam]

Remaining screenshots from previous reply.

[ihayesjr · Accepted solution]

The one thing I don't see in your command line backup is the -VAL parameter. This will validate the backup and the filestore and tell you if there are some files missing.

As an alternative to validating the backup, you can run a validation command to check for missing files in the file store without performing a backup. http://help.autodesk.com/view/VAULT/2018/ENU/?guid=GUID-DD35E4E0-C756-4B30-B276-B82268C0CEDE

You should run this to see if there are missing files.

If these indicate that you don't have missing files, something else would have to be investigated with your restore.

[milindnalgirkar]

Thanks for sharing the data. Some thoughts and comments:

1. How long does it take to copy the backup to the new servers?  About 8.5 hours: Why does it take this long to copy the backup to the new server? What sort of bandwidth do you have between old and new servers? Are you putting the backup straight on the new server or backing up local and copying it to the new server, is the E: in you backup script below mapped to something?

2.CD "C:\Program Files\Autodesk\ADMS Professional 2018\ADMS Console"

Connectivity.ADMSConsole.exe -Obackup -BE:\Backups\Full\20200623 -VUAdministrator -VPadminPasswd -DBUsa -DBPsaPasswd -LE:\Backups\Full\20200623\BackupLog.txt: The command seems OK-- how long does it take to run the backup with this command? Is E:\ a mapped drive or local?

3. Looks like the file store validation is not yet complete on the new server. Check in the ADMS Admin tool if the job to validate filestore is running, or refresh the ADMS console a few times to see if the count updates.  Just a side question -- did you upgrade SQL on the new server? Meaning is the SQL version on old and new server different? That might explain the reduction in Vault DB size.

Why the number of files are different can be different reasons -- refer to @ihayesjr comment -- perhaps one explanation is that the files in the old filestore were actually more than what was recorded in the Vault DB, ADMS runs a vacuum cleaner to clean the files that do not match with the DB records. It might not be a big issue, but you can investigate what has been going on.

[itsam]

1. I am backing up locally.  I am then using Robocopy to copy the backup directory from the old server (local) to the new server.  Both servers are in the same datacenter.  Both the old and the new servers have a separate storage drive local on each server as drive 'E:'.  They aren't mapped to anything else.

2. I don't know how long it takes to run this command.  I didn't run it as it was recommended not to.

3. The SQL versions on both server are the same.  I am running the command line validation now.  I do see some errors in the log files, but I'm not sure what they mean.  I've attached them.

[ihayesjr · Accepted solution]

Looks like it is timing out during the file validation.

https://knowledge.autodesk.com/support/vault-products/learn-explore/caas/sfdcarticles/sfdcarticles/H...

[milindnalgirkar · Accepted solution]

Your goal of the exercise is still to complete the server migration over the weekend right, so you dont have to do a delta/incremental migration? Just want to make sure we are still on the same page.

1. Why is the robocopy taking 8.5 hrs for a ~500GB copy on the same data center? Can you tell what the connection speed between the two servers is? You can also run a robocopy with the /mir option so essentially as the backup gets created it gets copied on the server. ROBOCOPY \\sourceserver\share \\destinationserver\share /MIR -- but you should not be needing this. See point #2 below.

2. I would suggest you run the command -- you will notice that the backup is way faster. Because now essentially ADMS will only backup the DB (About 50GB) and local copy the filestore (about 500GB) -- this has to be under an hour I think.

3. Do increase the time out settings on *both* the old server and the new server per @ihayesjr  recommendation. In addition to the restore, the backup maybe slow because it is timing out also -- check the backup log on the old server to see if you are seeing similar time out messages .

[itsam]

I have applied the timeout changes to both the old server and the new server as depicted by the KB article.

I initiated a backup on the old server with the command line.  The backup completed successfully (per the log file) and took almost 17 hours.  I've verified the  File Store sizes between the original and the backup and they are both 468 GB.

I am seeing the following error in the AVFS-log file:

Error: Soap Exception ( mesg-id = 637285772113255636 )
Exception: KnowledgeVaultDoesNotExist [133] (Vault)
Stacktrace:    at Connectivity.Server.Proxies.ServiceAccessor`3.tryexecute(String remoteServer, Action`1 action, Int32 attemptCount)
   at Connectivity.Server.Proxies.ServiceAccessor`3.Execute(String remoteServer, Action`1 action)
   at Connectivity.Filestore.AuthenticationService.CreateSession(WinauthCredentials creds, String dataServer, String knowledgeVault, LicenseMode licenseMode, Boolean isLicensed, Byte[] p, SecurityHeader oldSession)
   at Connectivity.Web.Filestore.v23.WinAuthService.SignIn(String dataServer, String knowledgeVault)
   at SyncInvokeSignIn(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

and I am seeing this error in the vlog file:

Error: Soap Exception ( mesg-id = 637285772113255636 )
Exception: KnowledgeVaultDoesNotExist [133] (Vault)
Stacktrace:    at Connectivity.Server.Proxies.ServiceAccessor`3.tryexecute(String remoteServer, Action`1 action, Int32 attemptCount)
   at Connectivity.Server.Proxies.ServiceAccessor`3.Execute(String remoteServer, Action`1 action)
   at Connectivity.Filestore.AuthenticationService.CreateSession(WinauthCredentials creds, String dataServer, String knowledgeVault, LicenseMode licenseMode, Boolean isLicensed, Byte[] p, SecurityHeader oldSession)
   at Connectivity.Web.Filestore.v23.WinAuthService.SignIn(String dataServer, String knowledgeVault)
   at SyncInvokeSignIn(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

I'm starting the copy of the backup to the new server.  I don't know the transfer speed, but I'll ask the server guys.

Thanks,

[ihayesjr]

I wouldn't be concerned about these errors during the backup. That database could have been locked at the time a different process tried to gain access to it. 

[milindnalgirkar]

17 Hours!! Wow --That is unusual. I would check the antivirus settings on the server first -- is every file being written to the backup folder being scanned? What is the antivirus setting? There are some other things to check -- but this is definitely out of order. Can you post your backup log here also post the specification of the server -- is it a VM? How many processors/cores are assigned to it? How much disk space do you have available? How much RAM?

The KVM does not exist in the AVFS log points to something else. Perhaps your users are trying to log into the Vault without entering/selecting the Vault name. Can you post a snapshot of your backup folder -- expand on the databases folder in the backup and take a snapshot of that.

[milindnalgirkar]

Remember you have ~1.7 million files in the backup - so if each file is being scanned by the antivirus that would explain the time taken. Please provide answers to my previous questions. Concur with @ihayesjr  the AVFS error could be any different things not related to the backup.

[milindnalgirkar]

Similarly when you copy to the new server -- check the antivirus setting on the new server -- are all files that are copied being scanned? That could explain the slow copy time also.

[itsam]

Here are the posts that you requested.  I'd like to also note that the original File Store folder has 434,901 folders while the backup folder has only 434,867 folders.  That's a difference of 34 folders.  There are users working, but is that common?

The anti-virus is controlled by our Cybersecurity team and I cannot make any changes to it nor can I disable it.

Server Specs - For both current and new File Store server

• Virtual Server
• Windows Server 2016 Standard x64 Bit
• Intel Xeon CPU E5-2680 @ 2.70 GHz (4 Processors)
• 18 GB RAM

I've attached images with Disc space, Folder Properties, and the backup log.

[milindnalgirkar]

I would not worry too much about the difference folders or files between filestore and backup -- there can be any number of reasons. Perhaps the users have deleted some files but the ADMS console has not cleaned the filestore yet? Perhaps @ihayesjr can comment on this behavior.

I strongly suspect that the antivirus is making the backup, copy and restore this slow. My suggestion is that you show/discuss this with the cyber security - 17 Hours to backup is painful. And that makes you think of doing an incremental/delta backup, which should not be required.

The server specs confirm to Autodesk requirements -- so that is good. To speed things up, I would show the backup and restore behavior to the cyber security team and perhaps they can make an exception when you are switching over to the new server?

Finally in the last screenshot where you see total number of records returned by your *.* search -- I suspect the filestore validation is not complete on the new server. When is the data on the new server from? I thought you are going to undertake the server migration in near future -- is the data in the new server current?

[ihayesjr]

Agreeing with @milindnalgirkar ,

1. The backup log doesn't show any files are missing from the file store. This is really good! This means that the records in the database have matching files in the file store.
   1. No need to worry about the 34 file differences you see in Windows Explorer.
   2. The restore should also indicate that there are no missing files.
2. The search results difference is mostly caused by the search indexes not being up to date yet because you performed a restore. These search indexes are rebuilt as part of the Restore process and will take some time to finish. This has nothing to do with the file store validation.
3. I also agree that Anti-virus scanning is contributing to the amount of time it is taking you to backup and copy the files.

[itsam]

All tests have been successful.  Just one question:  Can users check out files in advance of the backup and then check them back in after the restore?  Or will this create problems?

Thanks.