Can't render volume based project with GPU (MtoA-4.2.4-2022)

[futuremotionrecovery]

I have a project that just refuses to GPU render a scene I'm working on. It's a huge pain, because everything so much more slow on the CPU. I have a good GPU (RTX 2080 TI) and I want to use it.

I've uploaded the project here:
https://www.dropbox.com/s/rh2pfdvka6iwxx4/Volume%20Disp%20Troubleshooting.rar?dl=0

Here is the error I get in the script editor after failure:

# Warning: /aiVolume/aiVolumeShape: could not set BOOL parameter "disable_ray_extents" # 
// Error: [gpu] an error happened during rendering. OptiX error is: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: cudaDriver().CuEventSynchronize( m_event ) returned (700): Illegal address, file: <internal>, line: 0)
GPU 0 had 10271MB free before rendering started and 7353MB free when crash occurred
// 

Here's the Arnold log file:

00:00:00 834MB | log started Sun Sep 26 19:46:02 2021
00:00:00 834MB | Arnold 6.2.1.1 [fbf3b21a] windows clang-10.0.1 oiio-2.3.2 osl-1.12.0 vdb-7.1.1 clm-2.0.0.235 rlm-14.1.3 optix-6.6.0 2021/06/04 18:35:03
00:00:00 834MB | host application: MtoA 4.2.4 3c94e997 (fix-4.2.4) Maya 2022
00:00:00 834MB | running on DESKTOP-AIKP4FP, pid=42092
00:00:00 834MB | 1 x Intel(R) Core(TM) i9-9900KF CPU @ 3.60GHz (8 cores, 16 logical) with 65467MB
00:00:00 834MB | NVIDIA driver version 471.68 (Optix 60801)
00:00:00 834MB | GPU 0: NVIDIA GeForce RTX 2080 Ti @ 1650MHz (compute 7.5) with 11264MB (10294MB available) (NVLink:0)
00:00:00 834MB | Windows 10 (version 10.0, build 19043)
00:00:00 834MB | soft limit for open files is set at 2048
00:00:00 834MB | 
00:00:00 834MB | [metadata] loading metadata file: C:/Program Files/Autodesk/Arnold/maya2022/plug-ins/mtoa.mtd
00:00:00 836MB | [volume] searching OpenVDB file: C:/Users/futur/Desktop/Volume Displacement Lite/cache/VDBs/cloud_1_sim_050.vdb
00:00:00 836MB | [volume] found grid "density"
00:00:00 836MB | [volume] found grid "fuel"
00:00:00 836MB | [volume] found grid "heat"
00:00:00 836MB | [volume] found grid "temperature"
00:00:00 836MB | [volume] found grid "vel.x"
00:00:00 836MB | [volume] found grid "vel.y"
00:00:00 836MB | [volume] found grid "vel.z"
00:00:02 1332MB | [bifrost] C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost\arnold-6.2.0.0
00:00:02 1332MB | loading plugin: C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost/arnold-6.2.0.0/arnold_bifrost.dll ...
00:00:02 1333MB | arnold_bifrost.dll: bifrost_graph uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_object uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_multires_volume uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_multires_implicit uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_volume uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_points uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_implicit uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_polymesh uses Arnold 6.2.0.0
00:00:02 1333MB | arnold_bifrost.dll: bifrost_blocks uses Arnold 6.2.0.0
00:00:02 1333MB | loaded 9 plugins from 1 lib(s) in 0:00.00
00:00:04 1384MB | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenSplineTranslator.dll
00:00:04 1384MB | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenTranslator.dll
00:00:04 1470MB | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/usdProxyShapeTranslator.dll
00:00:23 1506MB | 
00:00:23 1506MB | releasing resources
00:00:23 1504MB | unloading 14 plugins
00:00:23 1504MB | closing hairPhysicalShader_shaders.dll ...
00:00:23 1504MB | closing lookdevkit_shaders.dll ...
00:00:23 1504MB | closing mtoa_shaders.dll ...
00:00:23 1503MB | closing renderSetup_shaders.dll ...
00:00:23 1503MB | closing xgenSpline_shaders.dll ...
00:00:23 1503MB | closing mtoa_ParticleInstancer_proc.dll ...
00:00:23 1503MB | closing mtoa_ParticleVolume_proc.dll ...
00:00:23 1503MB | closing xgenSpline_procedural.dll ...
00:00:23 1503MB | closing xgen_procedural.dll ...
00:00:23 1503MB | closing alembic_proc.dll ...
00:00:23 1503MB | closing cryptomatte.dll ...
00:00:23 1503MB | closing synColor_shaders.dll ...
00:00:23 1502MB | closing usd_proc.dll ...
00:00:23 1500MB | closing arnold_bifrost.dll ...
00:00:23 1500MB | unloading plugins done
00:00:23 1500MB | Arnold shutdown

The most relevant bits of information I think is the CUDA error:

Error: [gpu] an error happened during rendering. OptiX error is: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: cudaDriver().CuEventSynchronize( m_event ) returned (700): Illegal address, file: <internal>, line: 0)

I have no idea why, but the versions of loaded bifrost dll's are on an older version of the Arnold core:

00:00:02  1332MB         | loading plugin: C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost/arnold-6.2.0.0/arnold_bifrost.dll ...
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_graph uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_object uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_points uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_polymesh uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_blocks uses Arnold 6.2.0.0

I have the latest MtoA 4.2.4 (6.2.1.1)
https://docs.arnoldrenderer.com/display/A5AFMUG/4.2.4
Shouldn't bifrost be using the new version?

And finally the delayed loading of the extensions:

00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenSplineTranslator.dll
00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenTranslator.dll
00:00:04  1470MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/usdProxyShapeTranslator.dll

I would be overjoyed if someone could help me troubleshoot and fix this problem.

Thanks so much.

[patrick.schock]

Hi @futuremotionrecovery,

it seems that GPU rendering of aiVolume output from the Bifrost graph is still not supported. Please see here:

Arnold GPU / Volumes (Maya)

Arnold for Maya User Guide | Bifrost Graph

I don't know when it will be implemented, but you can vote for it here: Maya Ideas.

Best regards,

Patrick