Job processor transfer speed

[gunnar_larsson]

Hi experts.

About two months ago I upgraded a system from Vault Pro 2012 to 2014 (SP1).

At the same time all clients were replaced by new fast ones and installed with Product Design Suite 2014 + Vault Pro 2014 (SP1).

At the beginning everything was running beautifully, but in the last weeks the speed of the Job processor has plummeted to a level where it is not helping with work, but rather acting as an increasingly irritating obstruction. 

Now, large Inventor dwg:s does not process dwf creation correctly because of timeout.

Something is very wrong and I need some help investigating why. This has happened some time after the installation.

Here´s some data:

Vault Server = 2008r2 Std SP1, running on a VM on a Xeon5560 (2processors) @ 2.8GHz with 6GB ram.

It has 13.2GB free (of 60) on C:\ and 79GB (of 300) free on E:\, which are both SAS drives. E:\ contains the Vault DB and Filestore. 

It is running MS SQL 2008r2 for Vault:

Microsoft SQL Server Management Studio   10.50.2500.0
Microsoft Analysis Services Client Tools  10.50.2500.0
Microsoft Data Access Components (MDAC)   6.1.7601.17514
Microsoft MSXML      3.0 6.0
Microsoft Internet Explorer    9.0.8112.16421
Microsoft .NET Framework    2.0.50727.5477
Operating System     6.1.7601

The internal network is AFAIK of Gigabit caliber all the way and should be more than fast enough.

The Clients and the Job Processor machine are all up to date with 24GB ram, SSD etc.

The Job processor machine is a fairly new DELL T1640 with 16GB ram, Xeon E3-1270V2 @3.5GHz, 240GB SSD + 1.8TB storage disk.

It runs Windows 7 Enterprise SP1.

I did some tests and I think the main probem is the transfer speed between the Vault server and the C:\temp\{unique number...}  folder on the Job processor machine.

I tried probing the ehternet connections on both machines, and the best I can get when the jobprocessor is running, is some 26000 kbit per second throughput.

I tried changing the network card settings on both server and JP-machine, duplex, jumbo packet etc, but it didn´t improve things.

When I do a "Get" on the dwg-file I tested,  It and all children are all downloaded to the working-folder in about 2.5 minutes.

This is perfectly OK, considering the dwg with sub´s  contain some 5000 files and is about 2,8GB in total size..

BUT, When i submit the job of creating or updating a new dwf via the Job processor something seems to stop or interfere with the job transfer.

The folder in C:\temp is created, but the job stops after approximately 2 hours, having recieved some 623MB and 1023 files, it´s done that twice now.

The jobserver queue then says "The new job processor service is not responding." on the job. Sign of a timeout?

The processes are a mystery: the "Connectivity.jobprocessor.delegate.host" is the most active, but still "lazy" and wobbling intermittently between 0% and 12% CPU load, while not rising above 450MB memory (the machine has 16GB).

The network load is slowly wobbling between 0% - 12% (0% for the most part) on a 1GBPS network card through gigabit switches to the Vault-server´s gigabit card. That´s way below expectations I think.

It seems to me that I´m getting something like 4-5% oooomph out of this system and I can´t figure out why.

I would like to see something like 90%.

There is a way to get the job through, and that is to do "Update Locally" from the Vault Client" on the Job Processor machine.

If I do that, it does not timeout and the dwf is finally created after approximately 6 hours.(!!!)

This is creating the dwf, but it is also during that time stopping all other jobs like update properties etc in the queue, people are continuing work on files before they are fully updated, which in turn creates a lot of errors like "Non tip version" etc.  

This problem has to be fixed, and soon.

Please help.

BR.

Gunnar_L

[Olaf.Górski]

Hey Gunnar_L,

So you think the problem could be the transfer speed between the Vault server and the C:\temp\ folder on the Job processor machine. Did you consider a possible impact of antivirus? Disable it for test purpose and repeat your tests. Is there any difference in place?

[gunnar_larsson]

Hi Olaf.

Yes, I have considered it.

First thing I did was to uninstall it on the Job Processor machine and exclude all Inventor file types and processes on the Server.

It had no impact as far as I can tell. BTW, we are running MS Forefront.

I doubt it has anything to do with that.

The download to the Workspace when I do a "Get" on the file set is around 2,5 minutes.

The transfer of the same file set to C:\temp when I pass a dwf update to the Job Processor is around 6 hours.

This is a remarkable difference, and it makes the Jobserver funktion totally useless.

Now, I am in the process of uninstalling every update made, to both the Server and the Job Processor machine, after April 10th.

The last test after that shows a minute difference in transfer speed, but it´s still timing out.

[fbarrafato]

Hello, did you ever resolve this problem? We are experiencing much the same issue on our Vault Workgroup implementation, normal file transfers seem quick on check in/out, yet the performance of file transfers to the dedicated Job Processor is incredibly slow.

The dedicated job processor is a brand new Win 10 machine, SSD, Core i7 4.0 GHz, 1 Gig network connection, 32 GB ram, there is no limitation on the processor box, seems to be an incredibly slow trickle of files to the machine.

We are also in the midst of testing moving our vault file-store from a traditional disk SAN to a much faster SSD SAN, early tests in our sandbox environment show absolutely no difference in this specific function.

Any thoughts would be appreciated, long time lurker of the forums, first time posting!

Thanks! 

[gunnar_larsson]

The problem has been reduced, but in my opinion the way that the job processor works is still VERY far from good.

I am waiting impatiently for a reprogramming of the way the file set is sent (or fetched) to the job processor.

Here´s a workaround, from AD support that helps a little:

Check following on the workstation on which runs the job Processor:

• JobProcessor.exe.config in  C:\Program Files\Autodesk\Vault Professional 2014\Explorer
• Set the value from "100"/"500" to "1" in the lines
      <add key="InventorUpdateRevBlockCount" value="100" />
      <add key="InventorDwfPublishCount" value="100" />
      <add key="TrueViewDwfPublishCount" value="500" />

• Set the value from "90" to "180" in the lines
  <add key="RequestSendTimeout" value="90" />
  <add key="RequestReceiveTimeout" value="90" />
  <add key="HostServiceReceiveTimeout" value="90" />
• Windows Registry
  The Job Processor uses in the backround the Inventor ApprenticeServer. The Information between Vault and Inventor ApprenticeServer is defined about the Registrie - key {C343ED84-A129-11d3-B799-0060B0F159EF}
• Please check in the subkey "InprocServer32", that the default data exist and is correct.
• Is this not correct, start the application Inventor with 
  Right Mouse Klick  "Run as administrator".

//GL