GPU cluster dose not benifit much

[365549058]

Hello, I have setup a GPU cluster, both two computers are with two A6000ada GPU. So, I have 4*A6000ada totally.

These two computers are similiar like twins.

I can finish the 100% still frame render in 17 seconds with one PC, I expect it should be 8-9seconds with the cluster. However, it takes 14 seconds.

By the way, the VRED file was made by VRED2024.1, and the cluster runs with VRED2025.3. 

Is any suggestion to improve the performance of the GPU cluster?

[michael_nikelsky]

Usually this hints that you are running into a limit since the scene is too lightweight to fully utilize the GPUs. There is always some serial code between each image sample that cannot be run in parallel, things like scene updates, especially for motion blur, photon tracing and things like that. Additional to that, there is also the network communication that can be limiting although this should not be a concern for offline rendering since there doesn´t need to be an update on the render display.

In general however, the more complex your scene is the more benefit you will get from clustering. An exterior car shot with just one IBL lightsource will not benefit as much as an interior shot with multiple lightsources.

But it is difficult to say where the bottleneck is without knowing the scene. 

[365549058]

Hello, actually the scene is not too lightweight, on the other hand, it is too heavy.

But thank you very much for your hint, the bottleneck is really the GPU usage.

I checked again and again, finally discovered that if I activate the GPU raytracing without cluster, the usage of GPU RAM is 30GB on each GPU. When I use cluster, the GPU RAM usage on the master PC is 47.5GB, almost 100%, the  usage is 30GB on the slave PC.

I found out two ways to solve this problem:

Solution 1: disable the option "Use for Raytracing" of the compute node of master PC.

Solution 2: in the Preferences, set the Optimization Mode to be "Optimize Memory Consumption" instead of "Optimize Performance".

For me, the only Solution is option 2. It's the only way I can use all the 4GPUs.

But, is there any negative effect to set "Optimize Memory Consumption"?

I tested, the time is almost the same to finish the rendering to 100%, no matter "Optimize Memory Consumption" or "Optimize Performance".

[michael_nikelsky · Accepted solution]

Ok, that makes sense. The head node still has to have a copy of the scene in memory at the moment.
There are a few things you can do to improve this a bit:

First the option you have already mentioned, "Optimize Memory Consumption" should be used. The only drawback of this option is that switching variants or showing/hiding geometries is slower since the data will be removed from the GPU if it is not visible and therefore the acceleration structure needs to be rebuild once it becomes visible again.

What should also be enabled in the preferences is the Clear GPU Memory option, otherwise memory that is used by OpenGL already will not be freed when switching to GPU Raytracing.

With 2026 there is also an option to use on the fly texture compression but this will cause longer startup times (something we will improve in a future version to make this option more usable). In 2025.3 you can only compress textures using a python command but I would not recommend to use it as it can cause several other slowdowns.

Then it can also be beneficial to start the rendering in a specific order:
When you have started VRED first switch to GPU Raytracing and then disable the rendering under Visualization->Rendering. Then load the scene and after that start the Cluster (hint: For the localhost set the Network Speed to 40 Gigabit as this will improve network communication speed a lot. For the other computer set it to the speed the network card is capable of).  Once the cluster is initialized reenable Visualization->Rendering. This will save a bit of memory as well since there won´t be a full initialization of the raytracer on the head node. 

Memory wise it should be the same if you start GPU Raytracing, then start the cluster and after that load the scene  but it might cause some unneeded initializations that are then thrown away again, so this order should be the safer way to go.

For interactive rendering in the cluster also test out the Cluster image compression settings. Depending on your network they can have a significant impact on the rendering performance. For offline rendering this doesn´t matter though.

Kind regards,

Michael