Arnold GPU for Maya - Error

[gw950619]

Hello guys

I bought two used 2080ti for rendering and Nvlink.

so I did some tests on the company's project. the asset has 14x albedo + 14x specular + 14x roughness + 14x displacement, all 4k UDIM. and it has 60k+ poly faces, subdivision set up to 4.

and the end is not what I hoped for, it's so sad, the red line made me to doubt myself whether this investment is correct? should I sell the cards CPU and MB?

anyway, I record a video from beginning ( I forgot to set log files, when I record the video, so I did again after I upload the video)

https://www.youtube.com/watch?v=uDa6Wv0kOSg

2x 2080Ti FE + Maya 2020.1 + Arnold Core 6.0.2.1 1.

Error - 1

// Error: [gpu] an error happened during GPU update : Insufficient device memory //  

Error - 2

// Error: [gpu] an error happened during rendering : Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Assertion failed: "!targets.empty() : Buffer abandoned but no other devices are allowed to hold a copy", file: <internal>, line: 676) //

arnold.1 - Error.zip

[James_Grimes]

Hi G W,

I am in the same boat as you. I have 4 2080 Supers that I bought for use with Arnold GPU. I did some tests with all 4 GPUs and with 2 GPUs only with NVLink. I too ran into the Insufficient device memory error. My test consisted of twelve trees at a total of about 3 million polys. Arnold GPU choked, even with NVLink. The 2080 Super GPUs have 8GB of VRAM. I understood that Arnold GPU pooled the memory with NVLink. I guess that isn't true?

I did the same test with Redshift. Redshift handled it with no problem. I then increased the number of trees up to 48 for a total of approx. 12 million polys. Redshift still had no problem.

I really want to use Arnold GPU. But it isn't clear that the dev team is close to solving this problem.

[thiago.ize · Accepted solution]

It looks like you did 5 subdivision iterations? Does it run out of memory if you do fewer subdivisions? From your video, I see that the CPU memory goes from 8.8GB before the render starts, to a max of 18GB before the error occurs (So Arnold used about 10GB of CPU memory). My guess is that the bulk of this memory could be Arnold doing the subdivision on the CPU and then when it tries to copy it to GPU it finds that the GPU won't have space for all the data and so reports this error. As for how much memory the GPU has, it has close to 9GB free at the start of the render, so if CPU arnold has 10GB of geo, then it wouldn't surprise me if GPU arnold will run out. Keep in mind that nvlink does not currently double your gpu memory capacity for geometry.

Also, no need to pre-populate the cache before rendering (I actually never use it).

What would be helpful is to try the same render on the CPU, with at least info verbosity, and then look at the stats printed out at the end of the log file. How much memory was used by the CPU? If it's just a GB or so, then yeah, something is wrong. If the CPU used about 5 or more GB of RAM, then the GPU probably would have failed since at the moment GPU doesn't store data as efficiently as CPU.

[thiago.ize]

Like I said for G W, try this test on the CPU and see how much CPU memory is used. That might help to pinpoint where the memory is going. If you are using texturing, keep in mind that at the moment GPU arnold uses significantly more texture memory than CPU arnold (we expect that that to be fixed soonish).

[DeclanRussell]

Arnold GPU does pool the memory with NVlink. Its worth noting though that you can only pool the memory between the cards that are NVlinked together. So say we have you case with 4 2080 supers broken up into two pairs connected with NVlink. Then in total the maximum memory you have available in theory is 16GB, but in practice it's lower because not everything can be shared.

[gw950619]

I uploaded 2 videos

Test on the CPU and GPU, this time, it has log files

https://www.youtube.com/playlist?list=PLC-LdVgMRKzXSgw7gNMLJ0b_BDMaszeXC

4K Video download:https://drive.google.com/open?id=11IqOmfNZn8b9m9C4QURcsF20mzuF42_8

projects.zip

[gw950619]

in the first video, I tested twice. the first one I set subdivision iterations directly to 5, I got Error - 1. Then I restarted the software and tested the subdivision iterations from 2 to 4, as you can see, in subdivision 3, Arnold used 4.8G GPU memory, so when I change to 4, I got another Error.

I know GeForce NVLink doesn't double GPU memory, but I remember in some specific software, it does.

In my test, it just hit the wall over 10GB, so 11gb+11gb=11gb in Arnold?

[thiago.ize]

Thanks for confirming you were running out of memory. Each time you subdivide, you get 4 times as much geometry, so if 3 iterations was using 4.8GB of GPU memory, then 4 iterations would use up to 19GB of memory, which is more than each 11GB card can handle. nvlink does allow the sharing of some memory, but for now the rtx hardware acceleration requires that this data be replicated on each GPU. The non-rtx accelerated geometry, such as curves and volumes, should be shared across gpus, so there you will see a benefit from nvlink. Likewise, textures will hopefully soon also be shared.

[gw950619]

100% understand that. Thank you for your patience.

now I know It's not a good time to invest GPU rendering at present unless you have 48GB of GPU memory in one card.

I decide to sell my GPUs, CPU, and Motherboard. Then I'm gonna buy a 3970x, and switch back to CPU rendering.

[thiago.ize]

Yeah, if you really do need hundreds of millions of triangles and lots of textures, then 8GB likely won't work. We do plan on reducing the amount of memory that triangles occupy on the GPU so it matches better the CPU, but even if we get a 2x reduction, that won't prevent you from doubling the number of triangles and being back to running out of memory.

If you add a third card for general purpose usage and then only use your dual 2080s for Arnold, that will give you an extra 3GB/card which might make the difference here. Try rendering on 1 GPU (the unused one) to see if it helps.

[thiago.ize]

Followup: Right now it looks like Arnold might have a bug that is causing us to report an out of memory error without factoring in the sharing done by nvlink. What's more, while some of the data can't be shared across nvlink, other parts can be, so when we fix this I suspect there's a chance your test might just pass (it'd still be close).

[gw950619]

Yes, 2080Ti is not enough even 3080Ti, so I already sold my CPU MB and GPU, 3970X is on.