Arnold GPU Render Times Increasing Every Frame

Arnold GPU Render Times Increasing Every Frame

ks172295
Explorer Explorer
132 Views
5 Replies
Message 1 of 6

Arnold GPU Render Times Increasing Every Frame

ks172295
Explorer
Explorer

I'm running into this issue so often now, I've seen a couple of threads about it but no concrete answers.

When I render a sequence, Arnold GPU seems to take progressively longer to render each frame. In this example, the sequence starts at around 90 seconds, but by frame 140 it's reaching 6 minutes to render a single frame! Why are these render times creeping up like this? Is it a memory leak? I don't think it's scene specific as I've had this issue on multiple scenes. It's super frustrating and makes me want to trade it out for Redshift in my pipeline.

0 Likes
133 Views
5 Replies
Replies (5)
Message 2 of 6

peter_horvath
Autodesk
Autodesk

I'm seeing pretty consistent render times in my test scenes, although they are not particularly complex or heavy. Behavior could depend on many things, including the Arnold version, graphics card, driver version, etc. Maybe a specific object or setting in the scene is triggering it. To get more detailed information on the issue, could you please render a sequence with Arnold debug log enabled and send us the log files? You can setup logging in the render settings on the Diagnostics tab.

peter_horvath_0-1756297158879.png


// Peter Horvath
// C4DtoA developer
0 Likes
Message 3 of 6

ks172295
Explorer
Explorer

The scene is relatively heavy - lots of geo, 4k textures and lots of lights, but no sims or volumes. I will do another render tonight with the debug log enabled and get back to you with the log file. Cheers 

0 Likes
Message 4 of 6

ks172295
Explorer
Explorer

Debug Log files attached!

I rendered the same sequence last night with the same result, massively inconsistent, growing render times. I also noticed they seemed to shrink again at the roughly 8 hour mark. Just super inconsistent. As an example, F106 took 4:09 to render, whereas F108 which is practically identical, took only 1:31.

0 Likes
Message 5 of 6

thiago.ize
Autodesk
Autodesk

It looks like you might be running out of GPU memory. When this happens Windows will move some of the data off the GPU. This is nice because you don't crash. But it makes things way slower. The log files show how much GPU memory is available at the start of the render and then how much we were able to get during the render:

log 0106
00:00:00 22030MB | GPU 0: NVIDIA GeForce RTX 4070 @ 2490MHz (compute 8.9) with 12281MB (7074MB available) (NVLink:0)
00:04:02 32138MB | peak GPU memory consumed 6958.21MB

log 0108
00:00:00 25612MB | GPU 0: NVIDIA GeForce RTX 4070 @ 2490MHz (compute 8.9) with 12281MB (10126MB available) (NVLink:0)
00:01:24 31866MB | peak GPU memory consumed 10062.74MB

Notice how the 0106 render had 7GB available instead of 10GB from the 0108 render, and it uses pretty much all of that 7GB while the 0108 used all of the 10GB it had available. I suspect you probably need 12GB or so of available memory to properly fit everything on the GPU. I went ahead and graphed the info from all your logs and it does show a strong correlation with render time and amount of free memory at start of render:

render_time_memory.png

 

The next question is why does your GPU sometimes have 7GB free and sometimes 10GB? This could very well be a bug in Arnold, though that would be a bit surprising because I would expect a buggy Arnold to either always hold on to the memory, so available memory stays constant or progressively gets worse over each render. Instead it seems to fluctuate and then at a certain point (frame 108) it somehow resets itself and you're back to 10GB free. Another possibility is that this could be c4d consuming GPU memory for its own use. A third possibility is that you have some unrelated program also running that is consuming varying amounts of memory?

 

We maybe never noticed a slowdown because we have GPUs with more memory and never ran out of memory. But we should be able to detect this slow reduction in available memory. We'll check that out in case that lets us repro.

 

If you have multiple GPUs, dedicating a GPU for the renderer would be ideal since then it would have the full 12GB of memory to work with while the c4d and other applications can use the other GPU...

0 Likes
Message 6 of 6

ks172295
Explorer
Explorer

Thanks for looking into it. 

I usually close down any other programs and stop any unnecessary processes when I kick off a render to keep as much memory free as possible, so I'd be surprised if it was caused by something outside of C4D / Arnold. I'll have a deeper look when I'm rendering again later anyway. I don't have another GPU unfortunately. 

0 Likes