0% GPU usage during cluster rendering

[Neil_Cross]

I've set up a standard cluster, Vred Pro main client with Vred Core remote nodes on a LAN (all 2023.2)

When I execute a render, the cluster is clearly working as the render times (GPU RT) reduces significantly as I add nodes, but when I look at task manager the GPU usage is 0% on the remote nodes.  Weirdly, the temperatures on the GPU does increase during the rendering but I'm baffled as to why the utilisation is shown as 0%.

Not sure if the above image will come through clearly, but the remote desktop on the right is the "RPL" node in the cluster, it's 67% through the render but both the iGPU and the A5000 show 0% usage, but the temp on the A5000 went from 27C to 51C when the render started so it's doing the work, just not registering as activity?

[richardlevene]

Dont you have to use a proper gpu profiler to see the gpu activity vs viewing in windows taskmgr? Or was that only on older windows versions?

[michael_nikelsky · Accepted solution]

The task manager only shows 3D (DirectX/Vulkan/OpenGL) usage by default, not Cuda usage. On the professional GPUs you can switch it to Cuda manually but it won´t save that setting. There is also a usage monitor in the nvidia settings that is much more reliable.

No idea what to do on Geforce GPUs though, maybe there are some other tools out there that give you a proper usage reading.

[Neil_Cross]

What's confusing me though is the same render task shows 100% GPU usage on the primary machine with the Vred Pro client, but the slave running Vred Core is doing the exact same task showing 0%.  

And the Vray benchmark which is an explicit CUDA test shows as 100% usage in task manager:

If you don't think there's anything concerning to follow up on and this is expected behaviour with Vred I'll move on and just check these other profiling tools out 👍

[michael_nikelsky]

I would consider the task manager to be completely useless when it comes to any GPU usage with CUDA/Optix. As long as the nvidia usage monitor doesn´t show any issue there is no issue.

But if you can switch the Task Manager from 3D to Cuda you should see some activity there as well. Not sure if this is possible to do on a Geforce GPU though because.....reasons I guess.....

I am actually more surprised that you see any activity on the main machine.

[Neil_Cross]

Is this profiler/usage monitor that you would use the one which is part of the CUDA toolkit?

Either way I managed get CUDA to show up in task manager

For anyone finding this in the future, you have to disable hardware-accelerated GPU scheduling in Windows graphics settings to get the drop down option to show CUDA use.

I'll still try and find a better monitor, just don't particularly want to install the full CUDA toolkit just for that tbh unless it's available somewhere separately.

[michael_nikelsky]

The usage monitor is in the Nvidia control panel under Workstation->Manage GPU usage.

Interesting find with the GPU scheduling, didn´t know that.

[Neil_Cross]

Got it thanks, the Workstation menu in Nvidia Control Panel didn't appear when you're remote desktop into a client

[michael_nikelsky]

There is also the non-fancy but very powerfull nvidia-smi tool you can run from the command line. It can give you tons of information about the GPUs. The common line I use with it would be this:

"C:\Windows\System32\nvidia-smi.exe" --query-gpu=index,utilization.gpu,utilization.memory,memory.used,clocks.gr,clocks.mem,power.draw,temperature.gpu --format=csv --loop-ms=1000"

[richardlevene]

GPUz is a free simple option. Like CPU-z