Thin Client service unavailable Subscription Release 2014

[mdellunto]

Hi guys,

The new 2014 thin client looks nice but why so many problems?

Upgrading from Vault Pro 2013 to Vault Pro 2014 SR1 sp1 we are trying out the new thin client. Everything seemed to be working great other then clicking on the item master link and getting a pop up. Not sure if anyone can help with this?

 

That put aside I thought this would be a good time to get my very busy production team to test it out.

The next day when I went down to show them the new faceny flashing light thin client I get this error when trying to login.

embarressed but not shocked. 😉 (autodesk) Next I tired the old thin client.

From this point I started looking into the system logs and came across my very spammed event viewer.

All the problems seemed to have started here.

A IIS reset let me login but then soon after i get.

(the request failed with HTTP status 503: service unavailable.)

checking the event logs and i see this and i keep getting service is busy etc...

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 5/27/2014 9:38:03 AM 
Event time (UTC): 5/27/2014 1:38:03 PM 
Event ID: 12d1282c0bbd43d0840341d9cf6466b2 
Event sequence: 40 
Event occurrence: 2 
Event detail code: 0 

Application information: 
Application domain: /LM/W3SVC/1/ROOT/AutodeskTC-1-130456712120751013 
Trust level: Full 
Application Virtual Path: /AutodeskTC 
Application Path: C:\Program Files\Autodesk\Vault Thin Client 2014\ 
Machine name: localhost 

Process information: 
Process ID: 3844 
Process name: w3wp.exe 
Account name: IIS APPPOOL\AutodeskTCPool 

Exception information: 
Exception type: WebException 
Exception message: The request failed with HTTP status 503: Service Unavailable.
at Autodesk.Connectivity.WebServices.KnowledgeVaultSvc._KnowledgeVaultService.Invoke(String methodName, Object[] parameters)
at Autodesk.Connectivity.WebServices.KnowledgeVaultSvc._KnowledgeVaultService.GetVaultOption(String optionName)
at DataManagement.Application.Web.VaultUtils.AdministrativeSettings.GetVaultSettings(Connection vdfConn)
at DataManagement.Application.Web.Controllers.AdministrativeSettingsController.BannerImage()
at lambda_method(Closure , ControllerBase , Object[] )
at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
at System.Web.Mvc.MvcHandler.<>c__DisplayClass6.<>c__DisplayClassb.<BeginProcessRequest>b__5()
at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.<MakeVoidDelegate>b__0()
at System.Web.Mvc.MvcHandler.<>c__DisplayClasse.<EndProcessRequest>b__d()
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Request information: 
Request URL: http://localhost/AutodeskTC/localhost/vault/AdministrativeSettings/BannerImage 
Request path: /AutodeskTC/localhost/vault/AdministrativeSettings/BannerImage 
User host address: 192.168.0.55 
User: radu 
Is authenticated: True 
Authentication Type: Vault 
Thread account name: IIS APPPOOL\AutodeskTCPool 

Thread information: 
Thread ID: 7 
Thread account name: IIS APPPOOL\AutodeskTCPool 
Is impersonating: False 
Stack trace: at Autodesk.Connectivity.WebServices.KnowledgeVaultSvc._KnowledgeVaultService.Invoke(String methodName, Object[] parameters)
at Autodesk.Connectivity.WebServices.KnowledgeVaultSvc._KnowledgeVaultService.GetVaultOption(String optionName)
at DataManagement.Application.Web.VaultUtils.AdministrativeSettings.GetVaultSettings(Connection vdfConn)
at DataManagement.Application.Web.Controllers.AdministrativeSettingsController.BannerImage()
at lambda_method(Closure , ControllerBase , Object[] )
at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
at System.Web.Mvc.MvcHandler.<>c__DisplayClass6.<>c__DisplayClassb.<BeginProcessRequest>b__5()
at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.<MakeVoidDelegate>b__0()
at System.Web.Mvc.MvcHandler.<>c__DisplayClasse.<EndProcessRequest>b__d()
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Custom event details:

Now i am at a point where i am about to give up on this Subscription Release and start testing the normal version. Just wondering if anyone from Autodesk can make my day 😄

[mdellunto]

A bit more testing and i can't even use the heavey client. 

Also I can't access the administraion tab in the ADMS console by gonig to tools>Administration I get.

looks like the server is doing its own thing in server tasks

I have never seen a problem like this in any vault version from 2009 -2013.

I wounder what maintenace does? wasn't in 2013.

[mdellunto]

Looks like after disabling content indexing service the server started to work again. Guessing the re index in the server task list cripped the server?

The is a feature that my company needs and have been living with out it for awhile due to a bug in 2013.

[mdellunto]

I have also tried with a non Subscription release SP2. Exact same problem.

[zongjie.ng]

Hi mdellunto, I am looking into the issue you mentioned and will try to reproduce it over at my side.

To clarify, you were able to login previously and the only issue you get is when you try to access the Item master by clicking on the link provided in Thin Client, which prompted that error message that you have attached in the above post. Then subsequently, you were not even able to login to Thin Client, where you get the 503 error message.

So my question would be, are you facing both the issues mentioned or just the Item Master link error?

Thank you for your query and I look forward to your confirmation on the defects. I will get back to you as soon as possible, based on any findings I discover.

[mdellunto]

Hi zongjie.ng

First off I was able to fix the 503 error and all the errors causing vault to stop working. The way i was able to do this was by disabling content indxing. Now the vault is fully working. 

I still get the error from the item Master. From the forum post here it sounds like the same problem but no soution for it. 

http://forums.autodesk.com/t5/Vault-General/Vault-2014-Subscription-Release-1-Web-Client-Error-Illeg...

If I could get content indexing working again I would be happy with that.

So the issuse i am dealing with is just content indexing. You can forget I said anything about the Item master.

I have enabled and disabled content indexing 3 times so far and everytime i get the same problem. When content indexing is enabled All the system log errors and vault 503 and ADMS service not found message show up. If i turn of content indxing and restart the server its like nothing wrong ever happened. 

We had problems with our 2013 vault and content indexing we could not checkin our out files slow etc... we disabled content indxing and everything waws back to normal. I opened a support case and we where told that 2014 would fix the problem. Seems like it just got worse. You can check the case if u would like [CaseNo:08263884.]

[zongjie.ng]

Hi mdellunto,

Thanks for the confirmation. I have went through the other thread you've linked, and is it possible if you can provide us with the dataset backup so that I may do some verification on the cause of the error for item master? Thank you.

As for the content indexing, I will see what I can find out and will get my colleague to assist you on that issue.

Thank you for your patience.

[samuel.seah]

Ensure that the Autodesk Data Management Job Dispatch Service is started and running in Windows Services (services.msc) before enabling the Content Indexing Service. 

If that does not work, please provide us with the server and IIS logs from the following locations:

C:\ProgramData\Autodesk\VaultServer\Filestore

• Vlog-YYYYMMDD.txt
  
• ADMSConsoleLog-YYYYMMDD.txt

C:\inetpub\logs\LogFiles\W3SVC1

Regards,

Samuel

[mdellunto]

Hi Samuel.Seah,

I always make sure Autodesk Data Management Job Dispatch Serivce is running before doing anything.

Thanks for the reply. I have sent all the log files you requested. Please see CaseNo:09635629.

Zongjie.ng,

It may be a bit hard for me to send you the dataset we will have to talk more about it due to the size and customer content.

[samuel.seah]

It seems there are some issues with the Job Dispatch Service from your first post. Are those errors returning whenever the Job Dispatch Service is running?

Please post the logs here instead. Thanks!

Regards,

Samuel

[mdellunto]

Hi Samuel,

Yes the Job Dispatch Service is running all the time. The errors do happen when the service is running.

Here are the logs files.

[mdellunto]

Looks like this problem has been fixed. I don't want to speak too soon but i think this link has fixed my problem.

http://crackingthevault.typepad.com/crackingthevault/2013/08/error-when-starting-administration-in-a...

Also after installing Cumulative Hotfix 2 for Autodesk Vault 2014 SR1, SP1 errors are gong and everything is working. I am going to give it a bit more time and see if the server starts acting up again.

Cheers!

[mdellunto]

Looks like the hotfix did not fix the problem and inside the IIS manager. The AutodeskDMPool Keeps crashing/stopping.

So as it stands the AutodeskDMPool only stops when content indexing is enabled.

[samuel.seah]

Thanks for the logs. Our Product Support team will be able to assit you.

-Samuel

[mdellunto]

Thats great news. I can't wait to here back from them.

[mdellunto]

Well Autodesk told me to live without content indexing and closed my support case. Kinda a sad solution.(couldn't believe how fast they gave up)

Looks like you guys are my only hope now. 

So it looks like its just content indexing thats cause's this problem. Its the same problem as http://crackingthevault.typepad.com/crackingthevault/2013/week34/

But nothing that they suggest fixes the problem. Not even opening a support case. 

[jesseck · Accepted solution]

One direction you can go is to split the IIS application pool into different parts for each web application into its own Application Pool - when Autodesk introduced the "FileStore Service" model in 2014, I found that had to be done in some environments when troubleshooting.  Once you identify the application that is causing the crash, the next move I typically do is to escallate the "Run As" user of that particular pool from "AppPoolIdentity" to "Local System" so that Application Pool permissions can be ruled out.

Also, out of curiousity, do you have a remote file store, or is it local to the Vault server?

[mdellunto]

Hi jesseck,

It's the AutodeskDMPool keeps crashing. I will give a try setting it to "Local System" Here is a screenshoot of my app pools it looks like with the subscription release they have already done this a bit. The highlighted pool is the one that stops after a crash and the errors you see in my first post from IIS.

.

I do not have a remote file store but i don't keep the file store on the C drive it is on the N:\ drive. I also followed this post.

http://crackingthevault.typepad.com/crackingthevault/2009/04/permissions-required-on-filestore-folde...

The odd thing is it only does this when content indexing is enabled. But works fine and doesn't crash when it is disabled. 😞

[jesseck]

Some other changes I've made in the past were to disable the "rapid fail protection" feature on the Application Pool temporarily to see if the pool would keep running. If you have more than 1 application running in the pool (the number of applications are just to the right of where your screenshot ended) you could also move each application to its own pool, compartmentalizing the failures.

[mdellunto]

jesseck.

Thanks for the tips.  I will take a look and see what i can do. I will get back to you about how it goes. I will check up on it on the weekend and let you know. If not then Monday.

Cheers!

Thanks Again.