Multiple Soap Exception Errors During Vault Backup

[dpeters2]

Windows Server 2008 Standard with MS SQL Server 2008 Standard on HP Proliant DL360 G5.

Vault filestore is approximately 45 GB and databases approximately 1.8 GB.
Vault backups have long seemingly random lag periods of 3 to 7 hours.
During Vault backups I am seeing many (hundreds or thousands) of the following error in the server log:
Entire server log is attached also.
What is going wrong?

6/6/2009 4:07:53 AM *******************************************************************
Error: Soap Exception ( mesg-id = 633798580734250000 )
Exception: DatabaseLocked [201] (backup, KnowledgeVaultMaster)
Stacktrace:
Server stack trace:
at Connectivity.Core.DataAccess.DatabaseLocking.CheckSiteAccess(String databaseName, KnowledgeDatabaseType databasetype, ICollection`1 requiredPermissions)
at Connectivity.Core.DataAccess.DatabaseLocking.AllowSiteAccess(String databaseName, KnowledgeDatabaseType databasetype, SiteAccessError accessError)
at Connectivity.Core.DataAccess.ServiceManagmentSink.a(Object A_0, SiteAccessError A_1)
at Connectivity.Core.Database.TransactionContext.a(SiteAccessError A_0)
at Connectivity.Core.Database.TransactionContext.OnSqlException(SqlException e)
at Connectivity.Core.Database.SqlAccess.e(SqlCommand A_0)
at Connectivity.Core.Database.SqlAccess.ExecuteNonQuery(CommandType commandType, String commandText, Int32 commandTimeout, SqlParameter[] commandParameters)
at Connectivity.Core.DataAccess.Events.CleanupEvents(TimeSpan expiration)
at Connectivity.Core.Services.EventService.CleanupEvents(TimeSpan expiration)
at System.Runtime.Remoting.Messaging.Message.Dispatch(Object target, Boolean fExecuteInContext)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Connectivity.Core.Services.EventService.CleanupEvents(TimeSpan expiration)
at Connectivity.Core.EventMonitor.Scheduler.a()
at Connectivity.Core.EventMonitor.Scheduler.ExecuteScheduledEvents()
at Connectivity.Core.Services.JobService.ExecuteScheduledEvents()
at System.Runtime.Remoting.Messaging.Message.Dispatch(Object target, Boolean fExecuteInContext)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [1]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Connectivity.Core.Services.JobService.ExecuteScheduledEvents()
at Connectivity.Web.Services.JobService.ExecuteScheduledEvents() Edited by: dpeters@whoi.edu on Jun 6, 2009 11:56 AM

[Anonymous]

Hi,

In 2008 and later, the vault automatically locks
out other operations while backup is running [before it was necessary to
shutdown IIS entirely] - so these messages represent background jobs that are
prevented from running until backup is done. In later releases the vault is more
intelligent about not logging these exceptions, but they are definitely not
anything to worry about.

Paul Gunn

[dpeters2]

Hi Paul,

I am glad to hear that the error messages are normal.

I do still wonder why the backups take such a widely varying time to complete.

Don Peters

[Anonymous]

Hi Don,

It is hard to say what is accounting for the
differences. Backup is a very I/O intensive operations - really it is just SQL
making a copy of the databases and copies being made of the files themselves.
So given this I/O intensivity, it is possible
that factors like hard drive fragmentation could be at play. I guess one
question is where you are backing up to - if to a network location, there is
always the possibility of network traffic being a factor.

Paul

[dpeters2]

Paul,

The Vault Backup writes to a local drive, then a network backup backs up the whole machine periodically. I have theorized that the randomness of the backup times might be due to the network backup job fighting for disk read time...

Don P

[Anonymous]

You could be right there Don.

If you ran Perfmon during the backup you could
monitor things like IO activity - and possibly get a handle on what contention
is happening.

Paul

[chris_turner]

I'm in the process of migrating an old 2008 Productstream Vault to 2011 Vault Manufacturing... I am going theough ADMS 2010 first as you can't migrate from 2008 straight to 2011.

When doing the RESTORE of the old 2008 PS Vault to VM 2010, I am getting a lot of the same Exception: DatabaseLocked [201} messages in the vlog... Is this also nothing too much to be concerned about?

There doesn't seem to be any other obvious problems, but I would appreciate anyone elses input before I back this up and then migrate it to VM2011.

Thanks in advance.

[paul.gunn]

Hi -

Yes, the database locked messages are expected. These are actions that will be blocked until the backup is done.

Pauol

[chris_turner]

Thanks for the input Paul 🙂