topic Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023 in Arnold for Maya Forum

Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Wed, 06 Apr 2022 16:31:46 GMT

Hi!

I just purchased a second RTX 3090-card to GPU render with, using Arnold 5.1.1 in Maya 2023 and I'm getting really bad results. Rendering with 1x RTX 3090 card is currently way faster than using 2x cards. Even on really small scenes with only a couple of cubes and a physical sky.

11 sek for a scene with 1x card and 1 min 36 sek for same scene with 2x cards.

Does this have anything to do with me not currently having an SLI/NVLINK bridge or should that not matter? I'm having a hard time finding any information if a NVLINK is a requirement or not. I really hope it isn't considering how hard a NVLINK is to get a hold of.

I've tried rendering with Redshift for Maya and that gives me 2x the speed with the new card without SLI/NVLINK. As expected. Same using Blender and Octane.

I'm using the latest Nvidia driver (512.15).

Appreciate any help I can get.

Thanks!

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Thu, 07 Apr 2022 02:09:30 GMT

What about the opposite of the simple scenes you tested? On complex scenes does it perform better?

I would have thought that even without nvlink it'd have performed better and certainly not be 9x slower.

In the testing we've done in the past, adding a second 3090 without nvlink didn't make the renderer slower.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Thu, 07 Apr 2022 02:49:49 GMT

It's still rendering very slowly on larger and more complex scenes. And it doesn't matter if I pick a specific card manually to render with, both cards individually renders equally great but not together.

I've tried earlier versions of Maya (2020, 2022) with latest Arnold and older versions of Arnold (5.0.0.1) along with older Nvidia Drivers and the problem is consistent all through.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Thu, 07 Apr 2022 14:08:16 GMT

Could it maybe have something to do with the wattage from the PSU? I'm currently on a 1200W PSU which should be enough, but maybe just enough? I had some minor issue after installing the second card (blue screen and freezing), but can't tell if it has to do with the PSU or not. And it's only happened once. I've read online that 1200W should technically be enough but that 1600W if prefered to avoid potential power spikes from the cards. Could it be that Arnold maybe uses more power from the cards so it creates a potential bottleneck when it doesn't have enough? Not sure if that makes any sense at all. I'm grasping at straws here and I unfortunately don't have a 1600W PSU lying around to test with...

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Mon, 11 Apr 2022 08:01:39 GMT

So I just purchased and installed a 1600W PSU after calculating that my TDP load is around 1350W. So theoretically the 1200W I was using isn't enough, even though nothing really crashed. But unfortunately with the new 1600W PSU installed there were still no difference in render performance with 2x 3090 cards in Arnold. It's still as poor as it was before but works really great with double the performance using Redshift for Maya. As it should be.

I've ordered a NVLINK that should arrive in a week or so, but since this shouldn't make a difference other than helping with memory usage, I don't see how I could continue using Arnold unless I recieve some troubleshoot assitance from Autodesk.

I'll return with an update as soon as I receive the NVLINK.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Mon, 11 Apr 2022 19:39:23 GMT

Some things you can do before you get the nvlink:

Try to run two GPU arnold renders at the same time, each on a different GPU. If these take (almost) the same time to render as a single render with just one GPU, then I think it's safe to say that you don't have any power supply or other hardware issues.

Another thing to try is dual GPU rendering with adaptive sampling disabled.

Finally, if you can render with verbose logs that include the stats that are printed at the end of the render and upload here, that can also help us. Do make sure that log file has those stats.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Tue, 12 Apr 2022 13:21:01 GMT

Thanks for getting back to me.

I've tried having two Maya instances rendering the same scene at the same time with one unique GPU for each scene and that worked fine. Same render time as before on each scene when rendering with just one GPU.

I've actually always had adaptive sampling disabled when rendering with GPU because I never understood the benefit of it, but turning adaptive sampling on/off made no difference during testing.

There's a txt-file that I've attached to this message with Verbosity Level on Debug (arnold.log_2xGPU.txt). The two errors included at the end of the log were something I managed to fix by turning off all Bifrost-plugins and restarting Maya. But that didn't solve any render issues unfortunately. Render was still very slow. These were the two errors I fixed:

1: [mtoa] [%s] Could not find %s in search path %s

2: Bifrost procedural could not be found: %s

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Fri, 15 Apr 2022 02:33:46 GMT

Looks like the Render View doesn't produce any profile info. Your log (and I can reproduce it) has that empty:

00:07:12 4430MB | -----------------------------------------------------------------------------------
00:07:12 4430MB | top session self-times by category
00:07:12 4430MB | -----------------------------------------------------------------------------------
00:07:12 4430MB | top session self-times by node
00:07:12 4430MB | -----------------------------------------------------------------------------------

Do you mind repeating this with the "Arnold RenderView?" If you can include the profile file that could also be helpful (see https://docs.arnoldrenderer.com/pages/viewpage.action?pageId=71013068). Note for the profile you'll need to close ARV for the file to get written.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

christopher.brandstrom — Fri, 15 Apr 2022 14:12:40 GMT

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Fri, 15 Apr 2022 14:15:29 GMT

Sorry, previous post was accidentally done using my private account and not work account.

I've attached the new log file while using ARV. I tried attaching the .json profile file as well but that doesn't seem to be supported here so I included a Dropbox link to the file instead. Hope that helps:

https://www.dropbox.com/s/8dt1gd2g30673ie/arnold_profile_chris.json?dl=0

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Fri, 15 Apr 2022 18:03:05 GMT

Thanks. This at least confirms that almost all the time is indeed being spent in the GPU (the "GPU execution" in the profile) and it's not a bug or issue in at least the Arnold CPU code (CPU is still used for certain things even when GPU rendering).

In our testing in the past with engineers from nvidia, when using dual 3090s without nvlink we were not able to reproduce this type of slowdown. We also have other users that regularly use dual 3090s (no idea if they have nvlink) without issues. However, I was able to reproduce a slowdown (6x in my case with dual rtx8000) when I rendered a scene that contained nothing. With a sphere and a plane it's a 4x slowdown. "Real" scenes I get a speedup with dual gpu.

My initial testing seems to suggest a scaling issue when the scene is very simple and it seems to go away with more complex scenes. Maybe other users that report success with 3090s are rendering more challenging scenes where they do get a speedup? How much of a slowdown do you get on a real scene with complex shaders and lots of objects?

On our end, now that we know how to reproduce this, we'll see if we can speedup simple scenes. For your record, this is being worked on in our internal ticket ARNOLD-10827.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Sat, 16 Apr 2022 08:59:40 GMT

Hi Thiago

Thanks for having a look through my render info. Worth mentioning is that before upgrading to 2x 3090 cards I had 2x 2080 TI with NVLINK that worked great. No slowdowns at all.

What you're saying makes sense in some cases. I went back after reading your post and did a test with a much heavier scene. Here's the approx scene information just to make it a bit more readable:

- 480 objects

- 30 textures (various sizes)

- 25 materials

- 8 million triangles total

- No adaptive sampling

- Camera samples at 15

- 1x RTX 3090

- Aborted render after 10% completion

Total render time: 4 min.

Then I rendered the exact same scene again and aborted after 10% completion but with 2x RTX:

Total render time: 1 min 58 sek. GREAT! That cut render time down to half.

So like you said, that made a big difference in render time. That time having 2x RTX 3090 cards for a fairly heavy scene really helped which is really good to know. But then I did a test render with a scene that I would think is not overly heavy, but not too simple either with the following:

- 127 objects

- 1 texture (16bit 4096x4096)

- 24 materials

- 1,7 million triangles total

- No adaptive sampling

- Camera samples at 15

- 1x RTX 3090

- Aborted render after 25% completion

Total render time: 8 sec.

And then again I rendered the exact same scene but with 2x RTX 3090. Also aborted after 25%:

Total render time: 1 min 5 sek. Not so great.

So biggest issue then is to know when to GPU render using 1 card and when to use 2 cards it seems. One can of course always test to see what works best per scene, but Arnold has a tendancy to crash when you switch between using all cards or specific/individual cards so it becomes a bit of a process to restart Maya and reopen the scene each time.

Worth mentioning is that use Arnold for mainly a concept art at the moment, so for me speed and simplicity is paramount. Most of the times I work with objects without proper UVs so I use triplanar projections to speed things up. Not sure how much that affects GPU render speeds? Same with curvature node to mix between materials. I've noticed that has a big impact on render speeds compared to just using plain materials with textures. But finding out you can turn off "Abort on render" helped tremendously in regards to stability when rendering with GPU which was great. Or at least if felt like it did. I have a fairly good CPU as well (Ryzen Threadripper 3970X) to fall back to when rendering, but I've been really trying to find the absolute best and fastest options for GPU rendering.

I used to work in production as a character artist so I've been using Arnold for almost 10 years. I think the first time we used Arnold was for the movie 47 Ronin when Arnold was in early Beta, back in 2011. So Arnold has a big place in my heart today when it comes to quality and user friendlyness. I can't personally think of a renderer that delivers better handling for material nodes and overall end quality, which is why I'm really keen on getting the GPU rendering to work for a quick and dirty workflow. It's understandable that I'm not Arnolds target audience at the moment. But I rally do appreciate the assistance.

Let me know if you have any other ideas I can try to speed things up or if you want me to send over some other profile files. But knowing that 2 cards does indeed help in certain cases shows promise for me.

I'll post more when/if I get the NVLINK regardless if it makes any difference or not.

Thanks!

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Mon, 18 Apr 2022 18:30:20 GMT

Glad to hear the slowdown roughly matches what we found, where it is only with simpler scenes.

We think we know what is affecting the scaling on simple scenes, at least with the dual rtx 8000 we have available to test with. If it is this issue, I would expect nvlink to at least help somewhat, though probably not fully. After/if we can fix that (without needing nvlink), then it'd be good for you to see if that fixes it for you on the 3090s. Unfortunately we can't give an ETA for when or if this will get fixed.

I'm surprised you need to ignore errors to improve stability. I would suggest making a new post about that so we can try and resolve that issue.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

TheChrisBrand — Wed, 27 Apr 2022 10:48:19 GMT

Sorry for the late response. I never got an automated email notification on your reply.

Just wanted to let you know that I've now recieved and installed the NVLINK and again tried it with the very small scene I mentioned in my first post. The scene that I previously rendered in 11 seconds with x1 RTX 3090 card and 1 min 36 sek with x2 RTX 3090 cards, now renders in 7 seconds with x2 RTX 3090 cards. I've tried various other "smaller" scenes that previously didn't render as intended just to be sure, but now renders perfectly with x2 cards at almost double the speed.

So, extremely good news for me at least knowing that I don't necessarily have to change my worklfow so that I can utilize both cards with Arnold as a GPU renderer on all type of scenes for the moment. A big relief!

I'm trying to be on top of new Arnold updates since I'm not working within a big pipeline, but if possible please let me know if you'd like for me to try out any future updates that might specifically solve the affect with scaling on simple scenes that you mentioned. I know me buying the NVLINK wasn't wasted regardless, but it would be good to know if that problem is solved.

Regarding the errors I recieve, I will try to "catch them in the act" when they happen and see if it really has anything to do with me turning on/off Abort on Error. I'm not excluding that it might just be a perceived solution on my part, and not a proper solution. But I'll make a new post about it when/if it happes.

Re: Multiple GPU (2x RTX 3090 no SLI/NVLINK) slower render than with 1x GPU on Arnold 5.1.1 and Maya 2023

thiago.ize — Wed, 27 Apr 2022 18:26:27 GMT

Thanks for the confirmation that nvlink seems to almost entirely solve your problem 🙂 That supports what I was seeing in my testing, so now I'm extra hopeful that once we fix what we identified it should fix your non-nvlink slowdown as well.