Arnold batch render 3,000x slower than rendering a single frame.

[isaac_morhaim]

Seems like i'm seeing variations of this on this forum but not something quite as drastic. Regardless of CPU or GPU, the batch render takes 1 min to render a frame when the single frame render takes 2-3seconds. This is occuring on 2 machines. One with the amd 64 core chip set with 4xA5000's and the other machine with a i9 16 core with 2 A6000's.

It doesn't matter what is in the scene. It can be a high poly textured vehicle using UDIMM textures or a shaded cube and plane. haha.

Any thoughts? 

[Stephen.Blair]

The actual pixel rendering takes a minute, or is a minute to start the mayabatch process, load everything, and then start the actual pixel rendering?

We'd really need to see some Arnold logs.

A single-frame render means Render > Current Frame, with the latest MtoA?

[isaac_morhaim]

Hey @Stephen.Blair, greatly appreciate you reaching out. I want to help give you all the info you need. These are high end machines from Puget systems. I pulled the mayarenderlog.txt files. Scroll to the bottom as it seems to have all the history from over the years. I've also attached a mayarender log from my personal machine which is an i7 6 core with a a6000. As you will see, it performs as expected where it takes 2-4 seconds a frame to render rather than my workstations that take about a minute per frame (batch rendering). Redshift batches fine if that helps with anything at all on all machines.

Thank you again.

[Stephen.Blair]

---

@isaac_morhaim wrote:

Hey @Stephen.Blair, greatly appreciate you reaching out. I want to help give you all the info you need. These are high end machines from Puget systems. I pulled the mayarenderlog.txt files. Scroll to the bottom as it seems to have all the history from over the years. I've also attached a mayarender log from my personal machine which is an i7 6 core with a a6000. As you will see, it performs as expected where it takes 2-4 seconds a frame to render rather than my workstations that take about a minute per frame (batch rendering). Redshift batches fine if that helps with anything at all on all machines.

Thank you again.

---

Thanks for the batch render logs.

I looked at the logs from 30 Nov for the rig_v4_arnold scene

• A single RTX A6000 takes 2s per frame
• The four NVLINKed RTX A5000s take 5s per frame (after a 10s first frame).

This is due to the not-so-great scaling on multiple GPUs, which is something we're working on. With four GPUs there's a lot of communication between GPUs to handle.

You'd probably be better off running four batch renders at the same time, one per A5000. Just remember to set the CPU threads to 16 (64 / 4) so that each render has enough CPU threads.

Also, I see you're using 7.1.0.0
We strongly suggest using the most recent Arnold, because that way you get all the bug fixes that have gone into the ten subsequent releases of Arnold. I don't think any of that will speed up the multi-GPU scaling, but it will avoid the other issues we fixed for customers:

7.1.4
ARNOLD-12590 - [GPU] Single channel 8-bit textures are too bright

7.1.2.2
ARNOLD-12345 [GPU] Disabling skydome_light interactively is ignored

7.1.2.1
ARNOLD-11064 - [GPU] Crash when rendering the N AOV with volumes

7.1.2
ARNOLD-12211 - [GPU] Optix denoiser memory leak when creating and destroying render sessions successively
ARNOLD-12177 - [GPU] Crash with ramp shader after aborting due to an error
ARNOLD-12166 - [GPU] Texture memory leak when creating and destroying render sessions successively
ARNOLD-12165 - [GPU] Incorrect volume bounds

7.1.1.1
ARNOLD-12253 - [GPU] Crash when deleting and creating render sessions for the same universe
ARNOLD-12037 - [GPU] Artifacts in Light Path Expression AOVs using closure labels

7.1.1.0
ARNOLD-12201 [GPU] Random slowdowns
ARNOLD-12017 [GPU] rendered output can contain random invalid pixels

[isaac_morhaim]

@Stephen.Blair Thank you so much for looking into this and giving me the insight into the situation. When you say "You'd probably be better off running four batch renders at the same time, one per A5000. Just remember to set the CPU threads to 16 (64 / 4) so that each render has enough CPU threads." you're implying that I manually break up the batch renders into 4 sets where you divide the frames by 4 so if there is 100 frames, batch 1 would be 1-25, batch 2 would be 26-50, etc? 

[Stephen.Blair]

---

@isaac_morhaim wrote:

@Stephen.Blair Thank you so much for looking into this and giving me the insight into the situation. When you say "You'd probably be better off running four batch renders at the same time, one per A5000. Just remember to set the CPU threads to 16 (64 / 4) so that each render has enough CPU threads." you're implying that I manually break up the batch renders into 4 sets where you divide the frames by 4 so if there is 100 frames, batch 1 would be 1-25, batch 2 would be 26-50, etc? 

---

Yes, that's just a suggestion. If you want to take advantage of all four GPUs for a render job and avoid the scaling across four GPUs, you could do what you just described.

[isaac_morhaim]

@Stephen.Blair Hey stephen, finally had a moment to explore  this option. It appears neither of these machines can render arnold scenes as intended even when you enable only one of the video cards under arnold/system/manual device selection. The machine is snappier in ipr mode with one device selected but in batch mode it still takes about a minute per frame rather than 3 seconds. Is this  where you set what video card gets used in a batch render or is it something else? This is the case in both machines loaded with a6000's and the a5000's. 

[isaac_morhaim]

@Stephen.Blair in addition to the above, it appears it's also an issue with arnold cpu rendering. On a new scene with a car that isn't textured but simply uses a standard arnold shader, it renders in about 3 seconds on the cpu when you render a still using the render current frame button. When you batch render, it's the same issue as the gpu, takes a minute to render a frame. 

[Stephen.Blair]

Render Current Frame is an IPR render that doesn't write to disk.

But it's still the same Arnold, nothing is different.

Restart Maya and do not do any render.

Set the log verbosity to Info.

Open the Arnold Render View (don't render, just open it).

Open the Log window.

Set the Log Frequency to Full

In the Arnold Render View, disable Progressive Refinement.

Render and get that log.

Let's compare that against a batch render.