Vault backup scheduled task not working after switching from Vault Basic to Pro

[edarlak]

What is the most common reason for the "Failure calling web services..." error when trying to run a nightly backup of vault via a batch file run with task scheduler?

I have two servers, as far as I know I setup Win server 2012 R2 and SQL 2014 the same exact way on each machine.  One has been running ADMS Server 2019 (Basic) for the last year, the new server is running ADMS Server 2020 (Pro), clients are now live on this server, everything is working except the nightly backup.  I have been logging in remotely every night and manually running a full backup through the ADMS console.  Really want to get the scheduled task working again.

I have attached a backup log file to this post, I redacted the server name and IP.  I also attached the bat file I am using for the backup.

On the server CMD is set to run as administrator, and the scheduled task is set to run with elevated privileges.

Is this an issue when moving from Basic to Pro, or am I missing some kind of setting in my default website or SQL?

While the server is local to our facility, corporate IT is located at another site and they control the VLAN for these servers, could it be some kind of setting on their VLAN or is this a localized server issue?

[johannes.bauer]

Hi,

it is very odd it is not working from command line, but working from your ADMS

Can you open ADMS / Tools / Administration without error?

Can you try open up http://xxxxxxxxxxxx/AutodeskDM/Services/Filestore/_impl/NotificationService.svc in a browser and see if that is returning a webpage?

One thing could be, that IIS needs to be started before you do the backup? can you try that?

also what happens when you executed "C:\Program Files\Autodesk\ADMS Professional 2020\ADMS Console\Connectivity.ADMSConsole.exe" -Obackup -B"C:\Vault_Backup\Vault\A" -VUxxxxxxxxxxxx -VPxxxxxxxxxxxx -S -LC:\Backup\BackupLog.txt manually when logged in with the account you are running the scheduled task?

Thanks

[edarlak]

Thanks for the suggestions.  This being a live server I cannot test immediately, but I can try those suggestions after hours.

I will say:

When logged in as the user that runs the scheduled task, if I manually run the .bat or manually execute the -O backup routine from command line, the same error happens.  I can watch windows explorer and see the backup folder being created, about 6.5 GB (out of 210 GB) of files are backed up before the error happens and the created backup folder disappears from the backup location.

I can open ADMS console just fine without errors, and perform a backup through the console without issue.

As far as IIS running, it is a live server so if I am correct, no clients could connect if IIS was not running. Clients can connect and use the vault just fine.

As far as navigating to the svc with a browser, see the attached image for the page that comes up when I do so.

[johannes.bauer]

Hi,

ok understood.

So when the backup fails - is there something else running at the very same time? Can you look at event log to see if you find something on the timestamp?

Maybe a sql restart, or a vmware backup or something which might interrupt the connection to either the sql server or the IIS.

Thanks

[edarlak]

Ok, so it seems the ADMS-[servername] account cannot login.  See the attached pdf.

The old server was setup on a "local" company domain, then part way through the year the server was moved to a corporate domain with much more stringent password policy.  The new server was setup the same way as the old, but on the corporate domain.

I don't know where exactly to change the ADMS-[servername] password. Do I do this in SQL Manager or ADMS console?  And should this password be the same as my domain login?  Very confused...  

Also, for my domain login, I have to change that password every 3 months, I am trying to avoid having to change a bunch of passwords on the server in order for the automated backup to run.   I already have to edit the scheduled task running the bat file every 3 months to update the password or the task will not run.

I guess what I don't understand is, the backup log shows a few databases being backed up, then I get the web error.  How would a couple databases get backed up if the ADMS account login failed?  How would the backup process even get that far along?

[johannes.bauer]

Hello,

that seems messed up.

If you click change on ADMS-TEST-VAULT-PRO it should update the password and set it into SQL Server.

Have you retried this operation?

Also maybe ADMS user has not access to all the databases you need to backup. When you click in login properties for ADMS-TEST-VAULT-PRO to user mapping, where does he have which access?

I would rather ask you to open a support case and look into this with product support.

Thanks

[edarlak]

I added my domain account to the SQL logins and the ADMS-[servername] password error went away in event viewer, however I still get the web service call error in my vault backup log.

My backup script stops IIS before the Vault backup and restarts after, some posts on these forums are stating that those IIS lines should be removed from the script?  This script has been carried over for many years, believe we started using Vault in 2005.

[edarlak · Accepted solution]

So we eventually solved the problem, however the solution does not make much sense to us, maybe someone can educate us?

Initially, we were unable to run the bat file manually, we tried running as admin, as well as setting CMD to always run as admin, didn't work, we'd still received the web services error.

So we went into IIS manager and looked at the AutodeskDM site, we noticed directory browsing was disabled, even though when I installed Windows Server 2012 R2 I had directory browsing enabled.  I looked on our ADMS 2019 server and AutodeskDM directory browsing was enabled, I do not recall ever explicitly setting this.  I'm wondering if this is a difference between installing ADMS 2019 Basic vs. ADMS 2020 Pro?

After we set the AutodeskDM site directory browsing to enable, we could run the bat script manually, and the backup would run fine.  We figured we were home free, so we setup the windows scheduled task to run the bat file as the same user with elevated privileges, come in the next morning to find the log file showing that calling web services failed again.

The only way we were able to resolve the problem was to REM out the starting and stopping of IIS in the backup script, this allows the task scheduler to run the command line backup without issue.

Can someone explain to me why there is a difference running the script as me logged in and double clicking on the bat file vs. "me" running the same script via task scheduler?  Is there some issue between task scheduler and IIS that I don't have when I run the script manually?