Serious Vault Stability Issues

[Rob.O]

Hi All,

I am hoping someone can help.  We installed Vault 2012 ADMS (basic) on our Windows 2003 64 bit server a couple of months back.  It was working great for the first 3-4 weeks. We tested it with a few users (4-5) prior to bringing it fully online.  About a month ago, we started adding users to support project workload as the Vault process was going well.  It seems that once once we started adding more users, the stability problems started (or it could just be a coincidence).  We currently have about 47 users with accounts.

Our Vault service freezes 2-3 times a day most days.  On occasion we can go 2-3 days without a single problem, then we have issues multiple days in a row.  We have had our IT consulting company working on the server to see what could be done to releive the problems, but after doing several things to maximize the server's performance, the problems are not resolved.

We have applied all Vault 2012 hotfixes, have removed several services from the server (such as Distributed File System Replication, etc...) and still we have issues.  Typically when the problem occurs, the entire server has to be shut down and restarted before the Vault service will run correctly again.

I have used previous versions of Autodesk Vault for 3 1/2 years without EVER having to restart the service or the server, so I know the software is very reliable.  I know there is a resolution out there, but I think I need the help of the experts to resolve it!

I have read that a defrag of the database may help to resolve these issues, so I will be attempting that tonight.  I followed the advanced config guide when doing the install, so most of the settings should be desirable.  Unfortunately, we had to install ADMS on a domain controller (backup) as we had no other option at the time.  We are willing to reformat the server and remove it as a backup domain controller if required to get this issue resolved, but only as a last resort.

I have attached the VLOG.txt file from June 28th.  It shows numerous errors that I don't know how to resolve.

Thank you greatly for any help you can provide!

[Rob.O]

Forgot to post server specs:

MS Windows Server 2003 R2

Standard x64 Edition

SP 2

Intel Pentium D CPU

3.00Ghz, 3.00GHz

4.00GB Ram

Thanks!

[Rob.O]

Something else I forgot to mention and possibly the determining factor in our problem.

The Vault service seems to freeze whenever the w3wp.exe service climbs above 280,000K.  On a fresh reboot, it usually sits just under 200,000K, but when the Vault freezes, the service is usually sitting above 400,000K.

Hope that helps someone determine what our problem might be.

Thanks!

[Kevin.Jaufmann-Ludwig]

Dear Dull_Blades,

for me it looks like you have a problem with the IIS.

Could you take a look at the windows event logs and post it to here if you see errors?

I hope we then see more details what is failing.

Other question: what version of SQL 2008 are you using and what service pack is applied?

[Rob.O]

Kevin,

Attached is the Application Event Log.  I noticed several errors related to Autodesk Vault and SQL, but have no idea what they mean.

Vault was installed on a system running Microsfot SQL Server 2008, but the Vault database is an Express edition.  See SQL version info below and database info attached.

SQL Version Info:

Microsoft SQL Server Management Studio10.0.1600.22 ((SQL_PreRelease).080709-1414 )

Microsoft Data Access Components (MDAC)2000.086.3959.00 (srv03_sp2_rtm.070216-1710)

Microsoft MSXML2.6 3.0 4.0 6.0 

Microsoft Internet Explorer8.0.6001.18702

Microsoft .NET Framework2.0.50727.3623

Operating System5.2.3790

We also just upgraded the server to 8GB of ram to try and help performance.

Thanks for your help!

[Rob.O]

Kevin,

A little more info (if it helps)...

Today the Vault service again froze.  We noticed that prior to the freeze, the memory of the W3WP.EXE process had climbed to 1,400,000K and that it was using only 10% CPU.

When the freeze occured, the memory of the W3WP.EXE process was about the same, but the CPU usage jumped to over 50%.  We restarted the "SQL Server (AUTODESKVAULT)" service and the Vault service was usable again.  The W3WP.EXE memory level was still as before as was the CPU usage (50%+).  Unfortunately, after about another 10 minutes, the Vault service froze again.

At the second Vault service freeze, the W3WP.EXE memory was still around 1,400,000K, but the CPU usuage spiked to 99%.  This time I ran IISRESET on the server.  The Vault service came back and the W3WP.EXE memory dropped to 80,000K and the CPU dropped to 0.  We have had no more freeze problems so far today.  The CPU usage has been running at 0% to 1% since the IISRESET.

This does seem to confirm that the problem lies in IIS, but I have no idea what the problem is.

Any ideas?

Thanks again for your assistance!

[hans.martin.haga]

WIth the server acting as a backup domain controller, Microsoft applies a set of group policies to the OS to lock it down and increase security. This creates problems for the ADMS-server, at least when running with the default settings from the installation.

With a Small Business Server 2003 (which HAS to be running as a domain controller), my solution to get a stable ADMS-server was to grant the AUTODESKVAULT account (impersonating user) administrative access on the server and changing the IIS default application pool identity to use the AUTODESKVAULT account.

[Rob.O]

Thanks for the tip!!!

We are in the process of trying a couple of other things to fix it, but if those don't resolve the issue, I will definaltey try your suggestion.

[Kevin.Jaufmann-Ludwig]

Dear Dull_Blades,

sorry for the delayed response here.

I can´t see the application event log here attached. I just see a screen about the SQL information's

Could you attach the logs please?

However not sure if it plays a role but you need to be aware that the Express Version of SQL has a limit of 1 GB memory allocation.

So your memory upgrade might only pay off when updating to a full SQL version.

Any news about the suggestion by Hans-Martin?

[Rob.O]

 Hi Kevin,

Sorry it took so long for me to reply.  I have been on vacation for the past several days.

We have had some small success in getting the Vault service to run more reliably, but we are still having problems with it freezing about once every 2 days or so.

We had the file store on a seperate server, so we moved that to the same server that ADMS and SQL are installed on.  That seems to have helped a little.

I have not attempted Hans-Martin's suggestion yet.  In fact, we are going to try and demote the server and remove it's backup domain controller status first to see if that helps.

Attached is the event log I attempted to post previously.

I will let you know how the performance is as soon as the server demotion takes place and we have some time to monitor the service.

Thanks! 

[lohrum]

I am having this exact issue but using completely different hardware, OS and Vault version.

At least 3 times a week, user's cannot connect or checkin /checkout drawings.  If we re-start the SQL services (SQL Server and SQL Server Agent), the vault frees up and we can use it again.

As "Dull_Blades" reported, the culprit is w3wp.exe*32.  It consumes large amounts of memory and then everything freezes.

My server environment:

dedicated Windows server 2008R2

SQL server 2008 standard edition

4 cpus

32 G ram

ADMS 2011

NOTHING ELSE IS RUNNING ON THIS SERVER!

Definately stability issues with SQL and IIS.  Any other suggestions on a fix?

[bjsmith50]

We are also experiencing this problem on mutiple Vault Pro 2011 systems.  This problem occurs mostly during periods of peak activity.  We have ADMS and IIS running on a separate server than our database.  It appears IIS will freeze and the IIS queue will grow and start consuming large amounts of memory.  Then an IIS reset will temporarily fix the problem.  

Our server environment:

Dedicated Windows web/filestore server 2008R2 64bit

Dedicated Windows SQL server 2008 (SP1) standard edition 64bit

4 cpus and 12 GB ram on each server

ADMS 2011 and IIS 7

We have tried everything we can think of.  Any other suggestions?

[jonkersuk]

Hi Guys,

Has anyone got to the bottom of this issue, we have exactly the same issues as described in the first post.

I have just tried what hans.martin.haga has suggested and made the local Autodeskvault user an Administrator, I have also changed the DefaultAppPool and the AutodeskAppPL pools to run as the intendity of the Autodeskvault user.

I will monitor over the next day and see if it performs any better.

Thanks,

Jon.

[mariepl]

Hi Jon

Was your monitoring conclusive?  We are having the same issues over here... (on Vault 2014)

Thank you

[lohrum]

We upgraded to VAULT PROFESSIONAL and have not had this issue since.

[mariepl]

Interesting... We are running Vault Pro as well, but the issue remains.

[ryan]

I did see an issue once with a customer whereby IIS/w3wp.exe would repeatedly crash.   Worked all day trying to get to the bottom of it and finally we discovered that the Vault Search Index had somehow gone haywire and was killing the process.  So, we just went into ADMS Console and did a Tools > Rebuild All Search Indexes (check the option to delete existing ones) and Voila, problem solved!  Have never seen it before or since but hey, live and learn.

[mariepl]

Thanks! will try that.