Maya 2020, Arnold GPU batch render stops after a few frames (Rendering completed)

[bhoyle]

Hello,

i´ve been working on car-rendering lately mostly doing stills via GPU which worked fine.
GPU: RTX 3080 (10GB)
Maya 2020.4
MTOA to 5.0.0.2

Simple Scene:
--------------------------------------
- 1 car (4M triangles)
- 1 HDRI Env.
- Shadow Matte Floor Plane
Car body uses 4K UDIMs - 3 sets - color/roughness/normal.
Car wheels use 4K Texture - color/roughness/normal
The few other textures are small game-textures.
--------------------------------------

Last night I wanted to render a turntable (PNG, RGB only, no AOVs) and Arnold GPU kept aborting the render after only 2 frames of 250 frames. It did that every time: "Rendering completed"
I updated MTOA to 5.0.0.2 and thought it was fixed but it only rendered more frames like ~ 20 and then still quit rendering. (one of the last three frames usually glitches with super weird texture effects, then renders 1 or 2 frames correctly and stops)

I looked into the Maya renderlog and the Error always mentions OptiX which is ON for denoising:
----

00:08:16 8490MB ERROR | [gpu] ran out of GPU memory during rendering. OptiX error is: Memory allocation failed (Details: Function "_rtContextLaunch2D" caught exception: Out of memory)

GPU 0 had 6374MB free before rendering started and 114MB free when crash occurred

Try simplifying your scene or running on a GPU with more free RAM to see if it solves the crash.

----

Things I tried:
- Rendersettings / System / Maximum Texture Resolution: 2048
- Rendersettings / Arnold Renderer / Textures / Max Cache Size (MB): 6144 (was 4096)
- Rendersettings / Arnold Renderer / Textures / Auto-convert Textures to TX & Use Existing TX Textures (toggle on/off)

None of that things fixed the problem.

I cannot simplify my scene since it´s already pretty basic with no other geometry than the car itself.
Why can Arnold render 20 frames but not all of them?
It seems with every frame the GPU memory use increases altought it´s the same textures and geometry for every frame(?).

Was super happy with the GPU speed/workflow but if it can´t do batchrender properly on a modern GPU, I wonder how renderers like Redshift/Octane (GPU only) deal with GPU memory handling with much more complex scenes.

Thanks in advance for the support.

[Stephen.Blair]

What if you batch render in chunks, like on a render farm?

eg -s 1 -e 10, -s 11 -e 20, -s 21 -e 30, ...

[bhoyle]

@ thanks, I am not familiar on how to do that but will look into it.

Could you please shed some light onto why the GPU memory gets "clogged" at all?
1. in regards to the described scene which I consider rather simple with not a lot of textures (even if some are 4k the problems also persisted with max texture resolution set to 2048)

2. in regards to the scene complexity/textures not changing over the course of the 250 frames. So I don´t understand why memory consumption grows with each frame... (it´s the same textures/geometry for all frames)

Thanks

[bhoyle]

additional questions:
I was of the impression that "Max Texture Resolution" under the GPU selection in the render settings would restrict the rendertime TX texture file generation to whatever limit I put here.
So that for example "128" would make all my textures come out blurry in the render because they get shrunk/rescaled to 128x128.

Since nothing changes to my renders at all, I wonder what this setting really means/does?

[Stephen.Blair]

On Windows, open a command prompt and run these commands:

set PATH=C:\Program Files\Autodesk\Maya2022\bin;%PATH%
render -r arnold -s 1 -e 10 /path/to/yourscene.mb

Is it CPU or GPU memory that goes up? How are you monitoring the memory usage? With the Arnold log or with a utility?
We just logged a bug where the unaccounted memory (in the Arnold log) was increasing for each frame when rendering a sequence of frames from inside Maya (eg Render > Batch Render or Render > Render Sequence)

[Stephen.Blair]

I would check the detailed texture stats, which show which mipmap levels are used. If nothing changes, then the lower mipmap levels were already being used.

[bhoyle]

It was GPU memory that´s going up. I wasn´t really monitoring it professionally - just looking at the graph from windows task manager (see screenshot of original post).

Together with the error-log from mayaRenderLog.txt it was clear to be a GPU memory problem that occurs after a certain amount of frames.

With MtoA 5.0.0.0 it was after 2 frames, with MtoA 5.0.0.2 it is now after ~ 20 frames.

(tried to check the Arnoldlog-File but didn´t find it in the folder specified in render-settings, so not sure where to find it)

[bhoyle]

where would I find this information?

So was my understanding of what "max texture resolution" does correct? (because then it doesn´t seem to work - no matter how low of a value I put in, my textures render crisp and clean which would not be possible if a value of for example "128" would resize all textures to 128x128 pixels before rendering...)

[Stephen.Blair]

For Max Texture Resolution:

My test 4k texture and its MIP-map levels:

C:\Users\blairs>%A%\bin\oiiotool -v --info C:\temp\test.1010.tx
Reading C:\temp\test.1010.tx
C:\temp\test.1010.tx : 4192 x 4192, 3 channel, half openexr
    MIP-map levels: 4192x4192 2096x2096 1048x1048 524x524 262x262 131x131 65x65 32x32 16x16 8x8 4x4 2x2 1x1

To see what MIP-map levels are used during a render, I set the Arnold log verbosity to Info.
The log is printed in the Maya Output Window (on Windows) and in the Arnold Log Window.

With no Max Texture Resolution, I look at the MIP-COUNT information to see the 2096x2096 level is used 289 times, the 524x524 level is used 81 times, ...

00:00:09  2146MB         |   Image file statistics:
00:00:09  2142MB         |         opens   tiles    MB read   --redundant--   I/O time  res              File
00:00:09  2145MB         |       3    1      385        9.0                      13.8s  4192x4192x3.f16  C:/Users/blairs/Documents/maya/projects/default/sourceimages/test/test.1010.tx  MIP-COUNT[0,0,289,81,15,0,0,0,0,0,0,0,0]
00:00:09  2146MB         | 

If I set Max Texture Resolution to 131, then I see 9 opens of the 131-131 MIP level:

00:00:04  2198MB         |       5    1        9        0.2                       0.0s  4192x4192x3.f16  C:/temp/test.1010.tx  MIP-COUNT[0,0,0,0,0,9,0,0,0,0,0,0,0]

[jpgalves]

"Stephen Blair

What if you batch render in chunks, like on a render farm?

eg -s 1 -e 10, -s 11 -e 20, -s 21 -e 30, ..."



I'm trying to do that, in commad line I'm writing:

render -s 1870 -e 1873 MYFILENAME.ma || render -s 1874 -e 1878 MYFILENAME.ma


but it doesn't start the second command if i do not press "enter" when the first command finishes execute.

[sinaseifee]

My solution is to add a new line after each command so the cmd executes it after the previous line has finished its job.

For example:

"C:\Program Files\Autodesk\Maya2023\bin\Render.exe" -s 1 -e 10 "MYFILENAME.mb"
"C:\Program Files\Autodesk\Maya2023\bin\Render.exe" -s 11 -e 20 "MYFILENAME.mb"

"C:\Program Files\Autodesk\Maya2023\bin\Render.exe" -s 21 -e 30 "MYFILENAME.mb"

...

You can generate this with a script. Here is an example in python that outputs the batch render commands in chunks of 10 each in a new line. You can run it on an online python ide. The output can be copied and pasted into cmd:

file_name = "MYFILENAME"
start_frame, end_frame, chunk_size = 1, 1000, 10
render_cmd = "\"C:\\Program Files\\Autodesk\\Maya2023\\bin\\Render.exe\" -s START_FRAME -e END_FRAME \"C:\\Users\\Foo\\Documents\\maya\\projects\\default\\scenes\\" + file_name + ".mb\""
chunks = [i for i in range(start_frame - 1, end_frame, chunk_size)]
revert_size = 1

if chunks[-1] != end_frame:
    chunks.append(end_frame)

render_cmd_chunked = (render_cmd.replace("START_FRAME", str(chunks[i - 1] + revert_size)).replace("END_FRAME", str(chunk)) if i > 0 else "" for i, chunk in enumerate(chunks))

print( "\n".join(render_cmd_chunked) )