Vault Pro 2012 Replication Errors

[jddrinkwater]

Hi,

We have recently set up multi site replication (2 workgroups) with Vault Pro 2012.  We're having issues with performance on the publisher (Publisher Vault grinds to a halt when the subscriber Vault is connected).  We've narrowed it down to the KVM as being the culprit.  It seems to get stuck in a loop, and gives the following error in SQL:

The process is waiting for a response from the query '{call sys.sp_MSmakegeneration (?, NULL,NULL,NULL,100)}'

The CC library and Vault database seem to be running fine.  We have made some changes to the default timeout settings in SQL, but haven't been able to fix this error.

Ping is successful between both servers.  Avg response is ~175ms.  

All the problems go away when we reinitialize the KVM publication from inside the SQL management studio, but within 24h we bog right back down.  If we disable the SQL agent, performance is immediately restored.  There is no apparent hardware bottleneck, task manager shows minimal RAM and processor usage.  We need some help getting this sorted out.

Publisher specs:

Dell Power Edge 2900

40GB Ram

2x Xeon Quad core, 1.66GHZ

SQL 2008 R2 SP2

Vault Pro 2012, Update 1

Any help is greatly appreciated.

Thank you,

John

[jparks_79]

Hi john,

We had this problem with almost every site deployment (5 sites from Houston to shanghai)

There is some configuration in the sql studio that causes the initial replication to all but kill the environement. It is very apperrent with sites that have high latency and low bandwidth. This message you recieve is the same we received. The machine will appear like it is running idle for hours or days, then magically start to process data. For us, we just had to wait...eventually the database completely replicated and everything ran fine. We had one site take 2 weeks, some sites only a couple days (25gb database). And the entire time, sql monitor shows the message you had below(The process is waiting for a response from the query '{call sys.sp_MSmakegeneration (?, NULL,NULL,NULL,100)}). As well as a message stating "Uploading changes to the Publisher".

How long have you had the subscriber site connected? Is it completely initialized? Can you log onto it and add files? Maybe it just needs more time. When you log into the ADMS console and check the replication status, what does it say?

What we found was that reinitiallizing only gives temporary help. If you have autodesk subscription, I would issue a support request. Support was able to help us with some recomended changes to our SQL environment that improved our replication. Will probably help you too.

I might be able to help a little, but i dare not recomened any chages to your sql environement. Let me know if you are able to get it resolved.

Regards,

Jeremy

[jddrinkwater]

Hi Jeremy,

Thanks for the advice, we initially set the server up on our local lan, then sent it fedex to its final home.  That was about a month ago, but there have been several disconnections, restarts and reinitializations in between.  I'll take your suggestions to our IT and see where we're at.  Thanks for your help!

John

[jparks_79]

John,

Did you import your cad data while it was local to your publisher or did you just let the database replicate? Have you had any bulk imports from either server since you shipped it to its home?

We only seen this behavior during bulk imports with autoloader or with major database updates (1000's of files or properties) something seems to happen in the database that stops almost all transactions until the replication is completed on big updates.

Anyway, I wish you luck. If you get a chance and you have the opportunity, turn the sql agent on over the weekend and see if it completes its replication. You can check in the sql replication monitor. If the message says "Waiting 60 second(s) before polling for further changes." then all is good.

[jddrinkwater]

When we started it was in a brand new, empty vault DB.  There are 3 DBs associated with this vault: Filestore, a CC Library, and KVM.  All three were allowed to replicate across on our LAN prior to the final installation.

We've been adding a fairly significant amount of data since then, but not massive (in my opinion).  Currently there are 5100 objects in project explorer, which is nothing compared to the 250,000 in our previous database.  Most of that is within a folder set to not replicate.

Both the CC library and the Vault DB seem to be fine (they sit at "waiting 60 seconds..."), it's only KVM which is giving us the error.

SQL has been running since the middle of last week.  The point of no return has been crossed and we have to share designs with our other office, so we're stuck.

[jparks_79]

Hey John,

Did you ever get this resolved?

[jddrinkwater]

Short answer is no.  

We believe it is simply a slow internet connection between the sites, but there is nothing practical we can do fix that.  We would reinitialize SQL, but the servers would fall further and further behind each other until the whole system came to a halt.  The other office is now simply logging in to our vault over the WAN.  The same speed issues are killing those guys, but there are fewer people at that office and we don't have a better solution so they just have to suck it up.  It's hugely inefficient.

I really wish there were some real-world benchmarks from Autodesk that we could refer to.  For example, when I transfer a large (1Gb) file from a server at my location to the other office, I see an average speed of about 30kbps.  Is this enough for Vault replication to be successful?  Is this fast enough to work if we invest in WAN optimization (Riverbed)?  I have no idea, and I can't justify investing any more time or resources into this project without a reasonable likelyhood of success.

I wish I had a better answer!

John

[jparks_79]

Hi John,

That really sucks Autodesk couldn't get you any help.

Did you see this white paper they did a couple months ago on the Riverbed accellerator? If not, maybe give it a read. They had some really good test results with file replication site setup. Even if you chose to not have a multi site setup, it looks having these installed, it could even improve your guys connection which are sitting remote.

I have seen multiple autodesk posts on this product, I really just curious to see if anyone has actually bit the bullet and actually put in place.

Anyway, good luck!

Jeremy