GPU Rendering Crashes only on VM - NVIDIA OpenGL Driver error code:6

[erencberoglu]

Maya 2022.3 with mtoa 5.0.0.2 and Arnold Core 7.0.0.1 is crashing due to and OpenGL driver error. Error code:6

also does not release the memory after the rendering done or cancelled, Only releases if it crashes or starts a new task. Looked it up but couldn't find anything beside NVIDIA's support link that is in the error itself. Doesn`t prompt the error every time

We tested the same software setup with couple different hardware setup and this only occurs with virtual gpu setup

Systems that are working fine

Intel i9 9900 - Quadro 4000 - Standalone workstation - Uses 6.4GB VRAM for rendering the scene

Threadripper 2950x - RTX 3090 & RTX2080ti - Standalone workstation (3090 11.2 GB - 2080ti 8.4GB individually for rendering the scene)

System that having issue

Epyc 7H12 - 16 Procesor Assigned - A40-8Q - 8GB assigned - VM - (Exceeds what is needed for Quadro 4000 and Crashes after reaching 7.7GB VRAM for 15-20 Seconds)

For sanity check no issues with CPU rendering
Also tried with MtoA 4.2.4 - Still Same issue

The Same scene exported as .ASS to katana(4.0v5) and rendered with using 7.4-7.7GB VRAM but fails to release the memory and crashes after start rendering for second time.Also in Maya if I drop the resolution to save some VRAM it completes the render but doesn`t release the memory until it crashes or restarts a new frame

https://nvidia.custhelp.com/app/answers/detail/a_id/3553

[Stephen.Blair]

Can you get an Arnold log? (verbosity level = Info) Both for good frames and crashes.
Enable file logging so that the file is there even after a crash.

Or export the scene to ass and kick it (then the log will be in the terminal).

[erencberoglu]

Quadro 4000 - Driver V:472.84
A40-8Q - Driver V:742.39
VM-1080p-Crash.txt
VM-720p_SecondFrame-Crash.txt
VM-720p_Success.txt
VM-Mtoa 4.2.4_720p-Success.txt
VM-Mtoa 4.2.4_1080p-Crash.txt

[erencberoglu]

Quadro4000-1080p_Success.txt
3090-1080p-Success.txt

[erencberoglu]

@Stephen Blair any idea/update ?