Re: Memory Access Violation - Crashing on Renderfarm

cjh18 · ‎04-20-2020

Hey everyone, I'm trying to render our university project from Houdini to Arnold using our render farm, but we keep getting a tonne of errors that say "Memory Access Violation". We have checked our scenes, and none of the warnings in the logs seem to exist at all. We only have this issue when doing one of the following.

Adding fur/groom using 'hairgen' to our lit scene. Adding lights to a scene with the 'hairgen' in. The lights are Arnold Lights.

We also get "segmentation error" as well.

We can render locally without issues, but it's too slow to render shots.

This is the render log:

=======================================================
Error
=======================================================
Error: Renderer returned non-zero error code, -1073741819, which represents a Memory Access Violation. There may not be enough memory, or the memory has become corrupt.
   at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage azg)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
   at Deadline.Nodes.NodeRenderThread.c(TaskLogWriter adq)

=======================================================
Log
=======================================================
2020-04-19 22:52:06:  0: Loading Job's Plugin timeout is Disabled
2020-04-19 22:52:08:  0: Executing plugin command of type 'Sync Files for Job'
2020-04-19 22:52:09:  0: Synchronization time for job files: 463.707 ms
2020-04-19 22:52:09:  0: Synchronizing Plugin Houdini from \\deadline.herts.ac.uk\DeadlineRepository10\plugins\Houdini took: 0 seconds
2020-04-19 22:52:09:  0: Done executing plugin command of type 'Sync Files for Job'
2020-04-19 22:52:09:  0: Executing plugin command of type 'Initialize Plugin'
2020-04-19 22:52:09:  0: INFO: Executing plugin script 'C:\Users\renderfarm\AppData\Local\Thinkbox\Deadline10\node\CTA-B12-REND09\plugins\5e9cc7cffdfebd3838f9f148\Houdini.py'
2020-04-19 22:52:09:  0: INFO: About: Houdini Plugin for Deadline
2020-04-19 22:52:09:  0: INFO: Render Job As User disabled, running as current user 'renderfarm'
2020-04-19 22:52:09:  0: INFO: The job's environment will be merged with the current environment before rendering
2020-04-19 22:52:09:  0: Done executing plugin command of type 'Initialize Plugin'
2020-04-19 22:52:10:  0: Start Job timeout is disabled.
2020-04-19 22:52:10:  0: Task timeout is 2700 seconds (Regular Task Timeout)
2020-04-19 22:52:10:  0: Loaded job: wb_sh1230_027 - /obj/render/spec (5e9cc7cffdfebd3838f9f148)
2020-04-19 22:52:10:  0: Executing plugin command of type 'Start Job'
2020-04-19 22:52:10:  0: DEBUG: S3BackedCache Client is not installed.
2020-04-19 22:52:10:  0: INFO: Executing global asset transfer preload script 'C:\Users\renderfarm\AppData\Local\Thinkbox\Deadline10\node\CTA-B12-REND09\plugins\5e9cc7cffdfebd3838f9f148\GlobalAssetTransferPreLoad.py'
2020-04-19 22:52:10:  0: INFO: Looking for AWS Portal File Transfer...
2020-04-19 22:52:10:  0: INFO: Looking for File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2020-04-19 22:52:10:  0: INFO: Could not find AWS Portal File Transfer.
2020-04-19 22:52:10:  0: INFO: AWS Portal File Transfer is not installed on the system.
2020-04-19 22:52:10:  0: Done executing plugin command of type 'Start Job'
2020-04-19 22:52:10:  0: Plugin rendering frame(s): 1002
2020-04-19 22:52:10:  0: Executing plugin command of type 'Render Task'
2020-04-19 22:52:10:  0: INFO: Starting Houdini Job
2020-04-19 22:52:10:  0: INFO: Stdout Redirection Enabled: True
2020-04-19 22:52:10:  0: INFO: Stdout Handling Enabled: True
2020-04-19 22:52:10:  0: INFO: Popup Handling Enabled: True
2020-04-19 22:52:10:  0: INFO: QT Popup Handling Enabled: False
2020-04-19 22:52:10:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2020-04-19 22:52:10:  0: INFO: Using Process Tree: True
2020-04-19 22:52:10:  0: INFO: Hiding DOS Window: True
2020-04-19 22:52:10:  0: INFO: Creating New Console: False
2020-04-19 22:52:10:  0: INFO: Running as user: renderfarm
2020-04-19 22:52:10:  0: INFO: Executable: "C:\Program Files\Side Effects Software\Houdini 17.5.229\bin\Hython.exe"
2020-04-19 22:52:10:  0: INFO: Argument: "C:\Users\renderfarm\AppData\Local\Thinkbox\Deadline10\node\CTA-B12-REND09\plugins\5e9cc7cffdfebd3838f9f148\hrender_dl.py" -f 1002 1002 1 -g -d /obj/render/spec "C:/Users/renderfarm/AppData/Local/Thinkbox/Deadline10/node/CTA-B12-REND09/jobsData/5e9cc7cffdfebd3838f9f148/wb_sh1230_027.hipnc"
2020-04-19 22:52:10:  0: INFO: Full Command: "C:\Program Files\Side Effects Software\Houdini 17.5.229\bin\Hython.exe" "C:\Users\renderfarm\AppData\Local\Thinkbox\Deadline10\node\CTA-B12-REND09\plugins\5e9cc7cffdfebd3838f9f148\hrender_dl.py" -f 1002 1002 1 -g -d /obj/render/spec "C:/Users/renderfarm/AppData/Local/Thinkbox/Deadline10/node/CTA-B12-REND09/jobsData/5e9cc7cffdfebd3838f9f148/wb_sh1230_027.hipnc"
2020-04-19 22:52:10:  0: INFO: Startup Directory: "C:\Program Files\Side Effects Software\Houdini 17.5.229\bin"
2020-04-19 22:52:10:  0: INFO: Process Priority: BelowNormal
2020-04-19 22:52:10:  0: INFO: Process Affinity: default
2020-04-19 22:52:10:  0: INFO: Process is now running
2020-04-19 22:52:10:  0: STDOUT: EnvControl: HOUDINI_USER_PREF_DIR missing __HVER__, ignored.
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         | log started Sun Apr 19 22:52:11 2020
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         | Arnold 5.3.0.2 [f7602f75] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/04/09 17:16:44
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         | running on CTA-B12-REND09, pid=9836
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         |  2 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (16 cores, 32 logical) with 65422MB
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         |  Windows 8 (version 6.2, build 9200)
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         |  soft limit for open files raised from 512 to 2048
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         |  
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         | [htoa_op] Begin registration ...
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         | loading plugins from C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/plugins ...
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   187MB         |  alembic_proc.dll: alembic uses Arnold 5.3.0.2
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         |  cryptomatte.dll: cryptomatte uses Arnold 5.3.0.2
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         |  cryptomatte.dll: cryptomatte_filter uses Arnold 5.3.0.2
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         |  cryptomatte.dll: cryptomatte_manifest_driver uses Arnold 5.3.0.2
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         |  htoa_mandelbulb_proc.dll: mandelbulb uses Arnold 5.3.0.2
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | loaded 5 plugins from 3 lib(s) in 0:00.00
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/cameras.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/lights.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/operators.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/options.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/shaders.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [htoa_op] Loading metadata file: C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold/metadata/shapes.mtd
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [rop_operators] Registering operators ...
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   188MB         | [rop_operators] operator registration done.
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   204MB         | [vop_shaders] Registering shaders ...
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   206MB         | [vop_shaders] shader registration done.
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   206MB         | [htoa_op] End registration.
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   206MB         |  
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   206MB         | releasing resources
2020-04-19 22:52:12:  0: STDOUT: 00:00:00   205MB         | Arnold shutdown
2020-04-19 22:52:14:  0: STDOUT: Detected Houdini version: (17, 5, 229)
2020-04-19 22:52:14:  0: STDOUT: ['C:\\Users\\renderfarm\\AppData\\Local\\Thinkbox\\Deadline10\\node\\CTA-B12-REND09\\plugins\\5e9cc7cffdfebd3838f9f148\\hrender_dl.py', '-f', '1002', '1002', '1', '-g', '-d', '/obj/render/spec', 'C:/Users/renderfarm/AppData/Local/Thinkbox/Deadline10/node/CTA-B12-REND09/jobsData/5e9cc7cffdfebd3838f9f148/wb_sh1230_027.hipnc']
2020-04-19 22:52:14:  0: STDOUT: Start: 1002
2020-04-19 22:52:14:  0: STDOUT: End: 1002
2020-04-19 22:52:14:  0: STDOUT: Increment: 1
2020-04-19 22:52:14:  0: STDOUT: Ignore Inputs: True
2020-04-19 22:52:14:  0: STDOUT: No output specified. Output will be handled by the driver
2020-04-19 22:52:14:  0: STDOUT: Driver: /obj/render/spec
2020-04-19 22:52:14:  0: STDOUT: Input File: C:/Users/renderfarm/AppData/Local/Thinkbox/Deadline10/node/CTA-B12-REND09/jobsData/5e9cc7cffdfebd3838f9f148/wb_sh1230_027.hipnc
2020-04-19 22:52:34:  0: STDOUT: Warnings were generated during load.
2020-04-19 22:52:34:  0: STDOUT: Error loading: C:/Users/renderfarm/AppData/Local/Thinkbox/Deadline10/node/CTA-B12-REND09/jobsData/5e9cc7cffdfebd3838f9f148/wb_sh1230_027.hipnc
2020-04-19 22:52:34:  0: STDOUT: Warning:     
2020-04-19 22:52:34:  0: STDOUT: /obj/assetShaders/guardHairs/red_Hairs:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/assetShaders/guardHairs/light_Hairs:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/assetShaders/Cornea/ramp_float1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "use_implicit_uvs".
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/materials/guardHairs/red_Hairs:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/materials/guardHairs/light_Hairs:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/materials/Cornea/ramp_float1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "use_implicit_uvs".
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/rim/shopnet/arnold_vopnet/ramp_rgb1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/blue_fill/shopnet/arnold_vopnet/ramp_rgb1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/orange_fill/shopnet/arnold_vopnet/ramp_rgb1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/bounce/shopnet/arnold_vopnet/ramp_rgb1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/face_rim/shopnet/arnold_vopnet/ramp_rgb1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "wrap_uvs".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/spec:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/shadow:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/scatter:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/reflection:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/utility:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/cryptomatte:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/deadline1:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "dl_limit_frames_to_node".
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "dl_frame_dependent".
2020-04-19 22:52:34:  0: STDOUT:              
2020-04-19 22:52:34:  0: STDOUT: /obj/render/lookdev:
2020-04-19 22:52:34:  0: STDOUT:              Skipping unrecognized parameter "ar_export_referenced_materials".
2020-04-19 22:52:34:  0: STDOUT: ROP type: arnold
2020-04-19 22:52:34:  0: STDOUT: Enabled log to console
2020-04-19 22:52:34:  0: STDOUT: Set verbosity to detailed
2020-04-19 22:52:34:  0: STDOUT: Rendering frame 1002
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         | log started Sun Apr 19 22:52:34 2020
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         | Arnold 5.3.0.2 [f7602f75] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/04/09 17:16:44
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         | running on CTA-B12-REND09, pid=9836
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         |  2 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (16 cores, 32 logical) with 65422MB
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         |  Windows 8 (version 6.2, build 9200)
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         |  soft limit for open files raised from 2048 to 2048
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         |  
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         | loading plugins from C:/Docs/houdini17.5/plugins/HtoA/htoa-4.0.2_r236e192_houdini-17.5.229/arnold\plugins ...
2020-04-19 22:52:34:  0: STDOUT: 00:00:00  1038MB         | loaded 5 plugins from 3 lib(s) in 0:00.00
2020-04-19 22:52:35:  0: STDOUT: 00:00:00  1127MB WARNING | [void __cdecl htoa::arnold::Node::pullParameter] parameter not found: transform_type (ar_transform_type)
2020-04-19 22:56:11:  0: STDOUT: 00:03:36 13151MB ERROR   | signal caught: error C0000005 -- access violation
2020-04-19 22:56:11:  0: STDOUT: ****
2020-04-19 22:56:11:  0: STDOUT: * Arnold 5.3.0.2 [f7602f75] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/04/09 17:16:44
2020-04-19 22:56:11:  0: STDOUT: * CRASHED in tbbmalloc (0x000000001180acab) at 00:00:00, pixel (0, 0)
2020-04-19 22:56:11:  0: STDOUT: * signal caught: error C0000005 -- access violation
2020-04-19 22:56:11:  0: STDOUT: *
2020-04-19 22:56:11:  0: STDOUT: * backtrace:
2020-04-19 22:56:11:  0: STDOUT: *  0 0x00007ffe37f0bf4e [ai        ]                                                                      
2020-04-19 22:56:11:  0: STDOUT: *  1 0x00007ffe37f0b1ef [ai        ]                                                                      
2020-04-19 22:56:11:  0: STDOUT: *  2 0x00007ffe893f53ac [KERNELBASE] UnhandledExceptionFilter                                             
2020-04-19 22:56:11:  0: STDOUT: *  3 0x00007ffe8c937100 [ntdll     ] memset                                                               
2020-04-19 22:56:11:  0: STDOUT: *  4 0x00007ffe8c91f306 [ntdll     ] _C_specific_handler                                                  
2020-04-19 22:56:11:  0: STDOUT: *  5 0x00007ffe8c9335af [ntdll     ] _chkstk                                                              
2020-04-19 22:56:11:  0: STDOUT: *  6 0x00007ffe8c894aaf [ntdll     ] RtlWalkFrameChain                                                    
2020-04-19 22:56:11:  0: STDOUT: *  7 0x00007ffe8c93231e [ntdll     ] KiUserExceptionDispatcher                                            
2020-04-19 22:56:11:  0: STDOUT: >> 8 0x000000001180acab [tbbmalloc ]                                                                      
2020-04-19 22:56:11:  0: STDOUT: *  9 0x000000001180c993 [tbbmalloc ]                                                                      
2020-04-19 22:56:11:  0: STDOUT: * 10 0x000000001180d3fb [tbbmalloc ] scalable_malloc                                                      
2020-04-19 22:56:11:  0: STDOUT: * 11 0x000000002033d119 [libGU     ] GU_CleanTools::gu_EdgeMap::entries                                   
2020-04-19 22:56:11:  0: STDOUT: * 12 0x00000000208ebb8d [libGU     ] GU_PrimPoly::removeEdges                                             
2020-04-19 22:56:11:  0: STDOUT: * 13 0x00000000208ece6d [libGU     ] GU_PrimPoly::removeEdges                                             
2020-04-19 22:56:11:  0: STDOUT: * 14 0x00000000208efdd2 [libGU     ] GU_PrimPolySoup::countMemory                                         
2020-04-19 22:56:11:  0: STDOUT: * 15 0x0000000024adf236 [tbb       ] tbb::interface7::internal::task_arena_base::internal_max_concurrency 
2020-04-19 22:56:11:  0: STDOUT: * 16 0x0000000024ad90c0 [tbb       ] tbb::task_scheduler_init::default_num_threads                        
2020-04-19 22:56:11:  0: STDOUT: * 17 0x0000000024ad8656 [tbb       ] tbb::task_scheduler_init::default_num_threads                        
2020-04-19 22:56:11:  0: STDOUT: * 18 0x0000000024ad393c [tbb       ] tbb::internal::thread_sleep_v3                                       
2020-04-19 22:56:11:  0: STDOUT: * 19 0x0000000024ad38bb [tbb       ] tbb::internal::thread_sleep_v3                                       
2020-04-19 22:56:11:  0: STDOUT: * 20 0x00007ffe899403ba [ucrtbase  ] o_exp                                                                
2020-04-19 22:56:11:  0: STDOUT: * 21 0x00007ffe8bee7e94 [KERNEL32  ] BaseThreadInitThunk                                                  
2020-04-19 22:56:11:  0: STDOUT: * 22 0x00007ffe8c8f7ad1 [ntdll     ] RtlUserThreadStart                                                   
2020-04-19 22:56:11:  0: STDOUT: *
2020-04-19 22:56:11:  0: STDOUT: * loaded modules:
2020-04-19 22:56:11:  0: STDOUT: *    0x00007ffe37bf0000  ai
2020-04-19 22:56:11:  0: STDOUT: *    0x00007ffe89370000  KERNELBASE
2020-04-19 22:56:11:  0: STDOUT: *    0x00007ffe8c890000  ntdll
2020-04-19 22:56:11:  0: STDOUT: *    0x0000000011800000  tbbmalloc
2020-04-19 22:56:11:  0: STDOUT: *    0x000000001aa30000  libGU
2020-04-19 22:56:11:  0: STDOUT: *    0x0000000024ac0000  tbb
2020-04-19 22:56:11:  0: STDOUT: *    0x00007ffe89920000  ucrtbase
2020-04-19 22:56:11:  0: STDOUT: *    0x00007ffe8bed0000  KERNEL32
2020-04-19 22:56:11:  0: STDOUT: ****
2020-04-19 22:56:16:  0: INFO: Process exit code: -1073741819
2020-04-19 22:56:16:  0: Done executing plugin command of type 'Render Task'

=======================================================
Details
=======================================================
Date: 04/19/2020 22:56:17
Frames: 1002
Elapsed Time: 00:00:04:11
Job Submit Date: 04/19/2020 22:51:11
Job User: XXXXXXXXXXX
Average RAM Usage: 13356046336 (20%)
Peak RAM Usage: 20436717568 (30%)
Average CPU Usage: 30%
Peak CPU Usage: 100%
Used CPU Clocks (x10^6 cycles): 4976449
Total CPU Clocks (x10^6 cycles): 16588161

=======================================================
Node Information
=======================================================
Node Name: XXXXXXXXXXX
Version: v10.0.28.2 Release (31a4a2e50)
Operating System: Windows 10 Education
Running As Service: No
Machine User: renderfarm
IP Address: XXXXXXXXXXX
MAC Address: XXXXXXXXXXX
CPU Architecture: x64
CPUs: 32
CPU Usage: 5%
Memory Usage: 17.5 GB / 63.9 GB (27%)
Free Disk Space: 51.436 GB 
Video Card: ASPEED Graphics Family(WDDM)

thiago.ize · ‎04-20-2020

This says it is crashing inside houdini code. Have you tried upgrading your version of houdini and htoa to see if maybe it's a bug that has since been fixed?

Anonymous · ‎04-20-2020

I had similar issue with an asset which had multiple fur nodes for diff diff body parts. I checked each fur node rendering separately and my goodness i found the issue on a fur node. I got it fixed by the hair n fur artist.

cjh18 · ‎04-24-2020

I managed figure it out literally a couple of hours after posting. It was indeed an error with one of the 'hairgen' nodes. We think the renderfarm was leaking memory and crashing during the fur calculations before handing it off to Arnold. It happened to occur for another project I was freelancing on with 'hairgen'. It made sense as we also had an issue with motion blur streaks appearing too. We narrowed that down to a 'resample' node, which was causing vertex point numbers to randomise. We fixed that with and attribute wrangle.

My solution was to force an ASS archive, then render using the Arnold client on the renderfarm, which seemed to work fine. The only downside was that we had to archive every frame for up to 6 render nodes depending on the shot, and uploading that is a major pain.