Autodesk Data Management Job Dispatch Errors in Event viewer

[shastu]

This morning when I came into work, I was told that our content center was not working.  So I logged into our server where ADMS console is installed.  I launched the ADMS console and noticed that there were no libraries attached to our Vault.  I then tried the Attach library option, but nothing was there.   At this point, I just wanted to be able to have something for my users to use so that they could continue to work so I restored a backup.  Our backup runs at 1:00 a.m. and the restore was successful so I at least know that the problem happened after the backup was created.  So all seems to be good now, but I don't want this to happen again so I started to investigate to see if I could determine what happened.  I looked in the Event Viewer under the Application Windows Logs.  At 1:05:28 I saw the first error with a source of "Autodesk Data Management Job Dispatch".  The EventData read like this:

JobService.ExecuteScheduledEvents() failure: http://localhost/AutodeskDM/Services/Filestore/_impl/NotificationService.svc 201"

I then continued to work my way up, and at 1:23:26 I saw another error,  the source this time was "MSSQL$AUTODESKVAULT" and under the General tab read like this:

The client was unable to reuse a session with SPID 57, which had been reset for connection pooling. The failure ID is 23. This error may have been caused by an earlier operation failing. Check the error logs for failed operations immediately before this error message.

Another second later was another "Autodesk Data Management Job Dispatch" error:

It was very similar to the first one:

JobService.ExecuteScheduledEvents() failure: http://localhost/AutodeskDM/Services/_impl/NotificationService.svc 109

At 1:24:50, 1:24:51, 1:24:53, and 1:24:54 I saw four more errors:

They were all very similar and read something like this:

SQL Server is not ready to accept new client connections. Wait a few minutes before trying again. If you have access to the error log, look for the informational message that indicates that SQL Server is ready before trying to connect again. [CLIENT: 172.16.0.162]

1:25:13 saw this:

JobService.ExecuteScheduledEvents() failure: http://localhost/AutodeskDM/Services/Filestore/_impl/NotificationService.svc
The request channel timed out while waiting for a reply after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

Any ideas of what the problem was?  Did we just have something as simple as a network issue immediately after the backup happened or is there something more serious going on and we are going to see this again tomorrow morning or even sooner?

[ihayesjr]

@shastu 

Those errors would not be related to you missing your Content Center Libraries.

The Job Dispatch Service runs server tasks locate in the Tools -> View Server Tasks dialog in the ADMS Console.

1. If you were only missing libraries, you could have re-attached the libraries without restoring a backup. 

2. If the libraries were missing in the ADMS Console, you can look in the Console log to see when they were removed.

Check your Backup script to see if there are commands that remove the libraries.

[shastu]

Thank you for pointing me to the logs.  That is exactly where I was needing to go.  As I said, I tried to re-attach the libraries but I could not find them.  It has been forever since I have tried that so maybe I just didn't remember where to find them or something.  I will have to experiment with that this evening.  Regardless, something still happened that caused this issue and the logs are helpful.  Yes, our script does detach and re-attach our libraries.  I am not sure why we did that as I assume I can do a backup without detaching and re-attaching the libraries.  If you agree with that, I will look into doing that.  Back to the problem at hand.  I compared this mornings log file with yesterdays and there is definitely some interesting things to look at.  The first thing I noticed is the time differences.  For example, yesterday, it only took 14 seconds from when the log started to get to the point of detaching the library where as this morning it took 21 minutes and 20 seconds to get to that point of the script.  I have no idea why that would be.  They should have been relatively the same.  Any way,  this morning when the script got to the point of attaching the library I see an error:

1/31/2024 1:28:17 AM Attaching database... 0/1

1/31/2024 1:30:28 AM *******************************************************************

Warning: Detaching database AI2021_Bradbury failed

Exception: ALTER DATABASE failed because a lock could not be placed on database 'AI2021_Bradbury'. Try again later.

First thing I think is odd is it said it was Attaching the database, but then the failure says "Detaching"

I have gone ahead and copied the logs into a word document and attached.  If you could look through it and tell me exactly what could have happened that would have caused this, that would be great.  I see that at one point it gives an error message about the sa login so I don't know why that would be either.

ALTER DATABASE statement failed.

[ihayesjr]

@shastu 

According to the log, the library is being detached after the backup? Why, that doesn't serve any purpose.

The backup finished at 1:18, backing up that library database.

Then a detach happened at 1:21.

The library listed is a custom library and I highly recommend that you back it up all of the time.

[shastu]

That is what I was thinking.  I can't for the life of me remember why that got put in the script as it was probably 20 years ago, but there is no reason I can think of to do that.  I back up the library every morning.