Community
Arnold for Houdini Forum
Rendering with Arnold in Houdini and Solaris using the HtoA plug-in.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

HUSK command line render usd file in CPU renderfarm Error

22 REPLIES 22
SOLVED
Reply
Message 1 of 23
amy_adams_aaa
908 Views, 22 Replies

HUSK command line render usd file in CPU renderfarm Error

I have a render farm and it only has CPUs, no GPU.

after I  execute husk.exe command to render a usd file. I already set use CPU to render ,and don't use any optix denoise.

I set husk verbose 9 to get very detail debug info.

Here is only what I get.

 

"C:\Program Files\Side Effects Software\Houdini 20.0.506\bin\husk.exe" -s /Render/rendersettings --make-output-path -f 1 -R HdArnoldRendererPlugin --purpose geometry,render --complexity high --stderr c:\husk_error.log --verbose 9 "s:/render.usda"

[11:02:47] CPU Genuine Intel(R) CPU @ 2.20GHz
[11:02:47] Architecture - AMD64: mmx xmmx sse sse2 cmov fpcmov ht sse3 ssse3 sse4.1 sse4.2 avx avx2
[11:02:47] System limit: 8192 file descriptors

amy_adams_aaa_0-1703733061411.png

 

then Husk crashed

after I installed a graphic card, and run the husk command again, it worked correctly.

So How to disable GPU detection when loading the Arnold Plugin?

 

I am using

htoa-6.2.5.0_r82d49d1_houdini-20.0.547,

htoa-6.2.4.3_rf7fb3a9_houdini-20.0.506 ,

htoa-6.1.3.3_rdb15014_houdini-19.5.368

both not work.

22 REPLIES 22
Message 2 of 23

Even I use husk.exe --list-renderers

Husk Crashed.

but after I uninstall the htoa, husk.exe --list-renderers works.

amy_adams_aaa_0-1703733786800.png

 

Message 3 of 23

I don't understand why you don't get more of the Arnold startup log. 

I also don't have a machine without a GPU, but if I use an invalid index for CUDA_VISIBLE_DEVICES (to simulate having no GPU), I don't get any crash.

C:\Users\blairs>set CUDA_VISIBLE_DEVICES=-1

C:\Users\blairs>"D:\programs\side_effects_software\Houdini 20.0.506\bin\husk.exe" --list-renderers
00:00:00   212MB         | log started Thu Dec 28 08:37:53 2023
00:00:00   212MB         | Arnold 7.2.5.0 [c31d778e] windows x86_64 clang-15.0.7 oiio-2.4.1 osl-1.13.0 vdb-7.1.1 adlsdk-7.4.2.47 clmhub-3.1.1.43 rlm-14.2.5 optix-8.0.0 2023/12/05 11:40:27
00:00:00   212MB         | running on MTLMJ0HKD4C, pid=47152
00:00:00   212MB         |  1 x AMD Ryzen Threadripper PRO 3955WX 16-Cores  (16 cores, 32 logical) with 65386MB
00:00:00   212MB         |  Windows 10 (version 10.0, build 19045)
00:00:00   212MB         |  soft limit for open files changed from 8192 to 2048
00:00:00   212MB         |
00:00:01   360MB         | [color_manager_ocio] default ocio.config found in D:\programs\htoa\htoa-6.2.5.0_re21740b_houdini-20.0.506\htoa-6.2.5.0_re21740b_houdini-20.0.506\scripts\bin\../ocio/configs/arnold/config.ocio
00:00:01   361MB         | loading plugins from D:\programs\htoa\htoa-6.2.5.0_re21740b_houdini-20.0.506\htoa-6.2.5.0_re21740b_houdini-20.0.506\scripts\bin\..\plugins ...
00:00:01   361MB         |  alembic_proc.dll: alembic uses Arnold 7.2.5.0
00:00:01   361MB         |  cryptomatte.dll: cryptomatte uses Arnold 7.2.5.0
00:00:01   361MB         |  cryptomatte.dll: cryptomatte_filter uses Arnold 7.2.5.0
00:00:01   361MB         |  cryptomatte.dll: cryptomatte_manifest_driver uses Arnold 7.2.5.0
00:00:01   361MB         |  htoa_mandelbulb_proc.dll: mandelbulb uses Arnold 7.2.5.0
#########################################################################################
#  HDARNOLD_DEBUG_SCENE is overridden to 'C:/temp/hdarnold_debug.ass'.  Default is ''.  #
#########################################################################################
00:00:01   381MB         |  usd_proc.dll: usd uses Arnold 7.2.5.0
00:00:01   381MB         | loaded 6 plugins from 4 lib(s) in 0:00.13
00:00:01   400MB         | [htoa_op] Begin registration ...
00:00:01   384MB         | [obj_procedurals] Registering procedurals ...
00:00:01   384MB         | [obj_procedurals] procedural registration done.
00:00:01   419MB         | [rop_operators] Registering operators ...
00:00:01   419MB         | [rop_operators] operator registration done.
00:00:01   430MB         | [vop_shaders] Registering shaders ...
00:00:01   434MB         | [vop_shaders] shader registration done.
00:00:01   434MB         | [vop_imagers] Registering imagers ...
00:00:01   434MB         | [vop_imagers] imager registration done.
00:00:01   434MB         | [htoa_op] USD shader translation registration
00:00:02   453MB         | [htoa_op] End registration.
00:00:02   453MB         |
00:00:02   453MB         | releasing resources
00:00:02   438MB         | Arnold shutdown
#########################################################################################
#  HDARNOLD_DEBUG_SCENE is overridden to 'C:/temp/hdarnold_debug.ass'.  Default is ''.  #
#########################################################################################
Render Delegates:
 - BRAY_HdKarma (Karma CPU)
 - BRAY_HdKarmaXPU (Karma XPU)
 - HD_HoudiniRendererPlugin (Houdini GL) - unsupported
 - HdArnoldRendererPlugin (Arnold)
 - HdStormRendererPlugin (GL) - unsupported

 



// Stephen Blair
// Arnold Renderer Support
Message 4 of 23

I tried using the CUDA_VISIBLE_DEVICES=-1 trick, but it didn't fix the problem.

It's weird because this method works fine on my other PC that has a GPU.

But in my case, I'm working with computers that don't have any GPUs – not even the basic Intel ones, which is pretty standard for CPU render farms.

The last version that worked for CPU Farm was "htoa-6.0.2.2_r01f7d08_houdini-19.0.561". After that, it looks like Arnold needs a DCH driver, and I think that might be why it's not working now.

amy_adams_aaa_0-1703812514953.png

 

Message 5 of 23

I even try Kick.exe --help, not works. also on renderfarm PC without GPU 

 

C:\Users\Administrator\htoa\htoa-6.1.3.3_rdb15014_houdini-19.5.368\htoa-6.1.3.3_rdb15014_houdini-19.5.368\scripts\bin>kick
Arnold 7.1.3.2 [2a385cac] windows clang-10.0.1 oiio-2.4.1 osl-1.12.0 vdb-7.1.1 adlsdk-6.3.1.44 clmhub-2.0.0.235 rlm-14.1.3 optix-6.6.0 2022/09/12 08:50:17
kick: No arguments. Try kick --help for a command summary

C:\Users\Administrator\htoa\htoa-6.1.3.3_rdb15014_houdini-19.5.368\htoa-6.1.3.3_rdb15014_houdini-19.5.368\scripts\bin>kick --help

C:\Users\Administrator\htoa\htoa-6.1.3.3_rdb15014_houdini-19.5.368\htoa-6.1.3.3_rdb15014_houdini-19.5.368\scripts\bin>kick -device cpu --help

C:\Users\Administrator\htoa\htoa-6.1.3.3_rdb15014_houdini-19.5.368\htoa-6.1.3.3_rdb15014_houdini-19.5.368\scripts\bin>

 

Message 6 of 23
thiago.ize
in reply to: amy_adams_aaa

If you install an nvidia driver without having a GPU or remove the GPU after installing the nvidia driver, does arnold then work? I'm trying to determine if the issue is lack of a GPU or lack of some nvidia software.

Message 7 of 23
thiago.ize
in reply to: amy_adams_aaa

I tested on a local Windows 11 machine that only has an intel GPU card and I don't believe ever had an nvidia GPU or software and kick works properly.

 

When you tested after adding the GPU, were you still connecting remotely or could it have been working because you had a direct connection with access to a display? I'm still curious to know if removing the GPU and instead connecting the monitor to the integrated motherboard GPU (if there is one) still causes issues.

Message 8 of 23
thiago.ize
in reply to: amy_adams_aaa

Since you were able to reproduce this with kick, directly in a command line, that thankfully removes deadline and htoa as culprits, which is great. You've already reported that Arnold 7.0.0.3 works and Arnold 7.1.3.2 does not. What about the versions in between? Does 7.0.1.0 (the version after 7.0.0.3) work? If we can better pinpoint the version that broke this for you, maybe that can tip us off on what we could have changed on our end.

Message 9 of 23
amy_adams_aaa
in reply to: thiago.ize

After installed Nvdia driver, and unplug GPU, Arnold won't work again.
Message 10 of 23
amy_adams_aaa
in reply to: thiago.ize

No, I always connect the render farm with RDP.
No display connected.
Message 11 of 23
amy_adams_aaa
in reply to: thiago.ize

HtoA 6.0.2.2 using Arnold 7.0.0.3, works!
HtoA 6.1.0.0 using Arnold 7.1.0.0, not working.
I think DCH driver type could be the reason. 

 

amy_adams_aaa_0-1704332377178.png

 

Message 12 of 23
thiago.ize
in reply to: amy_adams_aaa

Thanks for narrowing the version that broke this. There's not much that immediately looks suspect here except for some changes related to upgrading the optix denoiser to version 7.4. Since the optix denoiser is provided through the nvidia driver, that's why we required a driver that is new enough that it contained this update. The DCH is unrelated: NVIDIA stopped shipping standard drivers (the last update is from 2021), so we wanted to make sure users were looking in the right place for the latest nvidia drivers.

 

Can you try one more command on the newer/broken build of Arnold without the GPU connected:

   kick.exe this_is_not_a_real_file.ass -v 6 -dw

and let me know if it prints anything out? The "this_is_not_a_real_file.ass" is just that, a random name of a file that you don't have. It will of course error out saying it can't read the file, but I want to see how far we get on that before it fails.

Message 13 of 23

ok, I have tried Arnold 7.1.3.2 and 7.1.0.0

Nothing prints out.  

 

 

amy_adams_aaa_1-1704418986515.png

amy_adams_aaa_0-1704418645113.png

 

Message 14 of 23

If I run on older version like Arnold 7.0.0.3

Seems worked.

amy_adams_aaa_2-1704419238604.png

 

 

Message 15 of 23
thiago.ize
in reply to: amy_adams_aaa

Thanks for trying that -- Since nothing was output, I suspect this is probably failing in the AiBegin call which happens at the very start of a render. I created the internal ARNOLD-14585 ticket for further investigation on our side.

Message 16 of 23

Thank you, Hope to fix this issue in the next version.

Message 17 of 23
thiago.ize
in reply to: amy_adams_aaa

One other thing to try: What if you fully uninstall the nvidia driver, does it then work? In particular, all instances of nvml.dll located in your system would need to be removed.

Message 18 of 23
amy_adams_aaa
in reply to: thiago.ize

Uninstall nVidia driver did not work.

I also tried another render farm node, it's "clean", not install any Nvidia driver, also have the same problem.

Message 19 of 23
thiago.ize
in reply to: amy_adams_aaa

Thanks for confirming this isn't a bug in nvml.dll.

 

I cannot reproduce this on my end because my computers do not allow booting without a GPU. But I think all I might need is a crash dump file from you. If we're lucky, you already have this. Take a look in

  %LOCALAPPDATA%\CrashDumps

and

   C:\ProgramData\Microsoft\Windows\WER\ReportArchive

to see if you have a recent kick.exe or AppCrash_kick.exe_* file you can compress and send me. Also let me know what version of Arnold generated this crash.

 

If you don't already have that, can you enable dump files on one of the machines, run arnold so it crashes, and then send me the resulting dump file.

There are instructions for doing that in https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps, but essentially you ...HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps and in there create a new DWORD called DumpType and set the Data to 2. You'll probably want to undo whatever changes you made once you're done sending me the crash dump.

Message 20 of 23
amy_adams_aaa
in reply to: thiago.ize

OK

I enable the DumpType and set the Data to 2.

and I can find the dump file in "C:\ProgramData\Microsoft\Windows\WER\ReportQueue"

but not in "C:\ProgramData\Microsoft\Windows\WER\ReportArchive"

 

seems it's related to the  nvcuda64.dll

please see attach zip

 

amy_adams_aaa_0-1705472899198.png

 

Can't find what you're looking for? Ask the community or share your knowledge.

Post to forums  

Technology Administrators