Vault 2014 Backup Failure

[Anonymous]

Our backup script runs each day but we get an error after part of the backup has completed which terminates the backup process. The error is below:

1/27/2015 8:00:23 PM BACKUP PROGRESS: The server console is currently in the process of backing up a database set. Please wait...
1/27/2015 8:00:23 PM ==================================================================
1/27/2015 8:02:42 PM Backing Up Databases... 0/10
1/27/2015 8:02:46 PM Backing Up Databases... 1/10
1/27/2015 8:02:49 PM Failure calling web services on 'http://servername/'. See log for details..
1/27/2015 8:02:49 PM Connectivity.Application.VaultManager.ServiceException: WebServiceError [8005] (http://servername/) ---> Connectivity.Core.Exceptions.EncodableException: WebServiceError [8005] (http://servername/) ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://servername/AutodeskDM/Services/Filestore/_impl/NotificationService.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it ipaddress:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   --- End of inner exception stack trace ---

I renamed the servername and ipaddress in the error. Any ideas where to start with this? We are running the web viewer to view drawings and that is working with no issues.

Thanks

[olearya]

Hi There, 

Are you running this backup script froma remote machine?  Is this a replicated environment?

[Anonymous]

Hi Allan,

No on both counts. Running the script on the local vault server (virtual box Windows Server 2008 R2), not replicated.

Thanks

Wes

[olearya]

Hi Wes, 

Does the manual backup work?

[Anonymous]

Yes the backup runs with no issues when run through the server Console. I am using the same script I ran in Vault Pro 2013. Please see the script below:

"C:\Program Files\Autodesk\ADMS Professional 2014\ADMS Console\Connectivity.ADMSConsole.exe" -Obackup -B"E:\Vault Backup\A" -WA -DBSC -L"E:\Vault Backup\BackupLog.txt" -S

Thanks

Wes

[olearya]

Hey Wes,

Apologies if this is a silly question, did you remove the vault user name and password when you pasted or is that not in the script?  You need -VU and -VP switches to run.

[Anonymous]

Allan,

I pasted the entire script, as is. I know for a fact this script worked in our 2013 Vault without the VU and VP switches. I have added those switches to the script and will see if it runs properly tonight.

Thanks

Wes

[Anonymous]

Having the same error message for VPS2015 R2.

Using a back-up script with user and pw switches. 

18-2-2015 21:17:27 Failure calling web services on 'http://vault/'. See log for details..
18-2-2015 21:17:27 Connectivity.Application.VaultManager.ServiceException: WebServiceError [8005] (http://vault/) ---> Connectivity.Core.Exceptions.EncodableException: WebServiceError [8005] (http://vault/) ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://vault/AutodeskDM/Services/Filestore/_impl/NotificationService.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.207.1.28:80

Tried to run a -Ob2bmigrate. All it returned was that all vaults & libraries already had been migrated.

Can an IISRESET /STOP before & IISRESET /START after the -Obackup  statements be the cause of this problem?
Not tested the manual back-up from the console. That's on the to-do for tonight.

[paul.gunn · Accepted solution]

Hi,

I think you have identified the problem. IIS should be running during the backup - otherwise web service calls made during backup will certainly fail.

Paul

[Anonymous]

In Autodesk Vault 2013 the backup script was setup to run after an IISRESET /STOP and then after running the backup IISRESET /START was run.

For my 2014 Vault script, I removed both of those IIS commands from my script and now it runs perfectly.

Thanks very much

Wes

[Anonymous]

I'm happy it worked for someone, my problems are still here.

1 issue was with the writing rights to the backup location. I fixed that, tried a backup from the SQL Server and the backup to that location works.

However I unveiled a second layer of mystery. When I extract the creation of the backup folder from the backup script and put it into a script on the desktop of that server 2 things are curious. Double clicking, will execute the script, creating E:\BackUp\Vault. When I run the script with the run-as-admin option, nothing happens.

I then deleted the backup script from the task scheduler, as it was set to run under the administrator.

Created a new task to run from the account I am using for access. The following was found in the ADMSConsolelog:

21-2-2015 6:39:22 Backing Up Databases... 3/7
21-2-2015 11:49:38 *******************************************************************
Warning: Deleting backup folders content failed
Exception: Access to the path '604580_628124_16.iam' is denied.

21-2-2015 11:49:50 Connectivity.Application.VaultManager.ServiceException: Cannot open backup device 'E:\Backup\VaultBackup_2015_02_20_21_01_13\databases\AI2015_[COMPANY LIBRARY]_dbbak'. Operating system error 3(The system cannot find the path specified.).
BACKUP DATABASE is terminating abnormally. ---> System.Data.SqlClient.SqlException: Cannot open backup device 'E:\Backup\VaultBackup_2015_02_20_21_01_13\databases\AI2015_[COMPANY LIBRARY]_dbbak'. Operating system error 3(The system cannot find the path specified.).
BACKUP DATABASE is terminating abnormally.

Still this Operating system error 3 persists.

Are there other tests I can do to come closer to fixing the basic backup from the ADMSconsole?

[Anonymous]

Update: manual backup from the ADMSConsole from my user account has succeeded.

Now to get it to succesfully run from the task scheduler....

[paul.gunn]

Hi,

Based on the error, it sounds like the windows account SQL server is running under (by default the 'network service' account) lacks permissions to the E drive location. When you did a manual backup, did you back up to that same location?

Paul

[Anonymous]

Thanks Paul,

After several more tests, the manual backup from the ADMS Console works perfectly (for several user/domain accounts).

Yet the automatic backup through a script and the task scheduler keeps stopping after 3 databases.

Yes, the backup location (E:\Backup\) is the same. I manually editted the security of that drive to grant everyone full access.

On the drive where the SQL databases are (D:\SQL\Data\) there are some share folders with different right policies (D:\Engineering\).

Could it be that the SQL user account does not have enough rights to pull a copy from there? Because with full right to everyone on E:\ the placing of files should be without problems?
Can it be that it is an issue with that 1 single database?

I've tested backing up that database from the SQL mngmnt studio, logged on as the standard sa user. That works without problems.
Are there any other tests I could run to pinpoint the problem? Upping the security clearance for the network service build in account?

Here's the log for that event again:

28-2-2015 3:22:48 28-2-2015 3:22:48 -Obackup -B"E:\Backup\" -VU[username] -VP[password] -S -L"E:\Backup\VaultBackupLog.txt" -DBSC

28-2-2015 3:22:50 ==================================================================

28-2-2015 3:22:50 BACKUP PROGRESS: The server console is currently in the process of backing up a database set. Please wait...

28-2-2015 3:22:50 ==================================================================

28-2-2015 3:46:42 Backing Up Databases... 0/7

28-2-2015 3:53:18 Backing Up Databases... 1/7

28-2-2015 3:53:35 Backing Up Databases... 2/7

28-2-2015 13:00:38 Backing Up Databases... 3/7

28-2-2015 16:57:14 Connectivity.Application.VaultManager.ServiceException: Cannot open backup device 'E:\Backup\VaultBackup_2015_02_28_3_22_51\databases\AI2015_[CUSTOMER LIBRARY]_dbbak'. Operating system error 3(The system cannot find the path specified.).

BACKUP DATABASE is terminating abnormally. ---> System.Data.SqlClient.SqlException: Cannot open backup device 'E:\Backup\VaultBackup_2015_02_28_3_22_51\databases\AI2015_[CUSTOMER LIBRARY]_dbbak'. Operating system error 3(The system cannot find the path specified.).

BACKUP DATABASE is terminating abnormally.

at Connectivity.Core.Database.TransactionContext.OnSqlException(SqlException e)

at Connectivity.Core.Database.SqlAccess.ExecuteNonQueryInternal(SqlCommand cmd)

at Connectivity.Core.Database.SqlAccess.ExecuteNonQuery(CommandType commandType, String commandText, Int32 commandTimeout, SqlParameter[] commandParameters)

at Connectivity.Core.Operations.Backup.BackupDatabase(DirectoryInfo backupLocation, String dbName)

at Connectivity.Core.Operations.Backup.BackupKnowledgeLibraries(DirectoryInfo databasesDir, BackupContents contents)

at Connectivity.Core.Operations.Backup.Execute()

at Connectivity.Core.Services.KnowledgeMasterService.Backup(String& backupPath, BackupOptions backupoptions, ICollection`1 excludedLibraries, OnProgressDelegate onProgress)

at System.Runtime.Remoting.Messaging.Message.Dispatch(Object target)

at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

--- End of inner exception stack trace ---

at Connectivity.Application.VaultManager.ProgressCommand.Execute()

at Connectivity.Application.VaultManager.BURWizardDataModel.InternalBackup(TypeFullOrInc type, String backupPath, String& actualBackupPath, List`1 excludedLibs, Boolean ignoreNonreplicatedFiles)

28-2-2015 16:57:17 Connectivity.Request.Common.RequestLibrary - RequestRuntimeStop()

28-2-2015 16:57:17 Connectivity.Request.Vault.RequestLibrary - RequestRuntimeStop()

28-2-2015 16:57:17 Connectivity.Request.Foundation.RequestLibrary - RequestRuntimeStop()

28-2-2015 16:57:17 Connectivity.Request.Document.RequestLibrary - RequestRuntimeStop()

[paul.gunn]

Hi Erik,

That is a real puzzler. Temporarily bumping permissions might be a good test.

Other things to try:

- temporarily detaching the library and seeing if backup will otherwise succeed

- temporarily disabling any antivirus if there is anything running (these can sometimes have weird effects)

Paul

[chris.downing]

this also working for me on a tried and tested script.