https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055484#M956 <P>The GPU will still use more texture memory than CPU. This is a known issue and one that we're actively working on. Last I checked, nvlink does not double the texture memory available, though one of the future nvidia driver updates should I think fix that.</P><P>Yes, GPU loads on demand, but it currently loads much more texture data than CPU. For now you'll have to use the workaround of setting a max texture resolution in the render settings.</P><P>Hopefully with this new build you should be able to render on multiple GPUs with textures without problems, provided it fits in memory.</P> Tue, 04 Feb 2020 21:43:56 GMT thiago.ize 2020-02-04T21:43:56Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055458#M930 <P>4x 1080 Ti's... C4D R21 up-to-date, latest NVIDIA Studio Driver, latest C4DtoA plugin, here's the log...</P><P><A rel="noopener noreferrer noopener noreferrer noopener noreferrer" href="https://answers.arnoldrenderer.com/storage/attachments/5603-gpu-cache-population-log.txt" target="_blank">gpu_cache_population_log.txt</A></P><P>I've deleted the cache in AppData\Local\NVIDIA\OptixCache\arnold-6.0.1.0_driver-441.66, but each time I try to run it with all four GPUs in the system, it fails (see screenshot below for final crash entry)... I delete the cache before each new attempt.</P><P>If I run it with just one GPU in the machine, it completes successfully... with all four in, it's a guaranteed crash.</P><P>For what it's worth, I can render with all 4 GPUs no problem with Redshift, so the GPUs themselves are fine.</P><P>Would love to know if anyone knows why this could be happening and/or how to get around it</P><P><span class="lia-inline-image-display-wrapper" image-alt="5604-annotation-2020-01-07-154700.png"><img src="https://forums.autodesk.com/t5/image/serverpage/image-id/1046417i527323C07F2B31A6/image-size/large?v=v2&amp;px=999" role="button" title="5604-annotation-2020-01-07-154700.png" alt="5604-annotation-2020-01-07-154700.png" /></span></P> Tue, 07 Jan 2020 21:01:28 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055458#M930 Eden_Soto2 2020-01-07T21:01:28Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055459#M931 <P>We had multiple reports of similar issues, seems to be an issue with multiple GPUs. We're investigating. Thanks.</P> Thu, 09 Jan 2020 11:26:11 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055459#M931 peter.horvath6V6K3 2020-01-09T11:26:11Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055460#M932 <P>I'm going to restate the problem to make sure I have it right. Because the screenshot shows a crash during rendering. </P><P>But really the problem happens before the render, when the pre-population fails with this error:</P><PRE>[ERROR] &nbsp; &nbsp; internal error during render permutation 11/40:</PRE><P>So, GPU Pre-Population failed, and then when you tried to render using that GPU cache, the render failed.</P><P><BR /></P><P>Can you delete everything in AppData\Local\NVIDIA\OptixCache, not just that one folder?</P><P>Then try to pre-populate again</P> Thu, 09 Jan 2020 11:36:09 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055460#M932 Stephen.Blair 2020-01-09T11:36:09Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055461#M933 <P>I emptied the whole AppData\Local\NVIDIA\OptixCache folder and ran the command again... it crashed again. New log attached.<A rel="noopener noreferrer noopener noreferrer noopener noreferrer" href="https://answers.arnoldrenderer.com/storage/temp/5575-gpu-cache-population-log.txt" target="_blank">gpu-cache-population-log.txt</A></P> Thu, 09 Jan 2020 20:14:17 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055461#M933 Eden_Soto2 2020-01-09T20:14:17Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055462#M934 <P>Would love to get this resolved... I still haven't really been able to gauge speed differences between the GPUs and the CPU... I can get the IPR to sometimes work without crashing, but it will eventually crash C4D completely, sometimes even locking up the system requiring a hard reset</P> Thu, 09 Jan 2020 20:16:11 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055462#M934 Eden_Soto2 2020-01-09T20:16:11Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055463#M935 <P>If you skip the cache prepopulation and just render a simple scene using 1, 2, and 4 GPUs, does it work? The cache prepopulation is just there as a convenience, but not using it is ok if you don't mind the first few times you render scenes it might take a few minutes before pixels start to appear.</P><P>Knowing the answer to this will also help us pinpoint where the problem might be coming from.</P> Thu, 09 Jan 2020 21:41:02 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055463#M935 thiago.ize 2020-01-09T21:41:02Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055464#M936 <P>If your machine is locking up, that could maybe mean your power supply is unable to power your four GPUs and CPU. If you disable a GPU or two, does it now work? It's possible Arnold is putting more demands on all of your system which ends up tipping it over the power threshold.</P><P>Another possibility for a system appearing to lock up is that you ran out of CPU memory. If you run the task manager you might be able to see if that is happening.</P><P>Otherwise, system lockups can be indicative of a driver bug.</P> Thu, 09 Jan 2020 21:46:41 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055464#M936 thiago.ize 2020-01-09T21:46:41Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055465#M937 <P>Just last week I did a 3:00 Redshift render with all 4 GPUs that took 18 hours... not a single hiccup, the GPUs are fine and I’m on the latest NVIDIA Studio Driver that hasn’t changed since then... Have a 2990WX Threadripper with 128GB of RAM and a 1600W power supply... there’s plenty there to handle it</P> Thu, 09 Jan 2020 23:30:09 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055465#M937 Eden_Soto2 2020-01-09T23:30:09Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055466#M938 <P>The only way I can render with the GPU is if I just have one in there system... but check my reply to your other note below... the system is plenty capable, so it seems as though it’s something specific to Arnold rendering on the GPUs... CPU is perfectly fine </P> Thu, 09 Jan 2020 23:34:52 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055466#M938 Eden_Soto2 2020-01-09T23:34:52Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055467#M939 <P>I'm running into the same problem. With multiple GPUs (I'm running 4 RTX 2080ti's), they will use all the cards UNTIL you turn textures on. Disabling the Textures in the DEBUG window, renders fine. I've tried converting the textures to different formats (TGA/TIF/EXR/etc), with and without MIPs, etc. Nothing works, unless you want to ONLY run one card and see your textures in the render. Any update of when we might be able to see a fix?</P> Thu, 16 Jan 2020 03:55:49 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055467#M939 Anonymous 2020-01-16T03:55:49Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055468#M940 <P>I never tried all four when disabling textures... I'll try that today and see if I get the same results</P> Thu, 16 Jan 2020 13:52:39 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055468#M940 Eden_Soto2 2020-01-16T13:52:39Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055469#M941 <P>Thanks Eden and Dante for your detailed investigations! It looks like NVIDIA has been able to reproduce a driver bug where using 4 or more GPUs with textures will usually cause a crash. This bug could in theory also cause crashes when not using textures or using less than 4 GPUs, though we haven't yet been able to confirm how common this is. They're making good progress towards fixing it, so fingers crossed, it will hopefully get fixed in one of the upcoming driver releases.</P><P>In the meantime, you can render with 3 GPUs in order to drastically reduce the likelihood of a crash and we'll also investigate whether there are any workarounds we can do in Arnold.</P> Thu, 16 Jan 2020 23:59:37 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055469#M941 thiago.ize 2020-01-16T23:59:37Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055470#M942 <P>Fantastic news! Looking forward to getting a working solution so I can do some thorough testing</P> Fri, 17 Jan 2020 00:29:19 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055470#M942 Eden_Soto2 2020-01-17T00:29:19Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055471#M943 <P>Why do you need gpu cache population? Is it render faster? Arnold 6 is very snappy without it...</P> Sat, 18 Jan 2020 17:25:08 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055471#M943 Anonymous 2020-01-18T17:25:08Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055472#M944 <P>You don't need it. All it does is pre-compile some shaders into the gpu cache so that when the first few times you go to render after upgrading your Arnold or nvidia driver version, you don't need to wait a few minutes for the GPU shaders to be compiled since they are already in the cache. On the other hand, once you've rendered a scene, those shaders will now also be in the cache, so now there's now no need to pre-populate.</P><P>Personally, I don't usually bother to pre-populate.</P> Sat, 18 Jan 2020 18:05:43 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055472#M944 thiago.ize 2020-01-18T18:05:43Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055473#M945 <P>Yes, i think it is the old technology already... <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P> Sun, 19 Jan 2020 20:10:47 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055473#M945 Anonymous 2020-01-19T20:10:47Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055474#M946 <P>For me it’s been a way to test and determine why my machine crashes with more than one GPU in the system... that’s why I continue to run it... if the machine crashes in the IPR or during output render, I don’t have a way to figure out why without the log? At least I don’t know of any other way</P> Sun, 19 Jan 2020 22:18:19 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055474#M946 Eden_Soto2 2020-01-19T22:18:19Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055475#M947 <P>FWIW, no combination of GPU selection makes Arnold GPU render stable, so selecting only three didn’t help for me... I’m going to physically remove one and see if that will do it... but like I’ve said in another reply, I can render with all four GPUs in Redshift till the cows come home and it never flinches, which is why I think the issue is unique to Arnold in my setup.</P> Sun, 19 Jan 2020 22:23:48 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055475#M947 Eden_Soto2 2020-01-19T22:23:48Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055476#M948 <P>You can get logs from IPR and batch renders. Links to instructions for how to do this are on the right side of this page. Search for "Instructions for generating full verbosity log files are available for". Let us know if you need further help getting logs -- we LOVE logs, so happy to help with that!</P><P>We are getting several reports of crashes in the GPU cache pre-population, so it's possible that's triggering more aggressively a bug, possibly the driver bug mentioned earlier?</P> Tue, 21 Jan 2020 22:48:43 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055476#M948 thiago.ize 2020-01-21T22:48:43Z https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055477#M949 <P>Arnold 6.0.1.1, released today, should fix the multi-gpu texture hangs. This won't help with the gpu cache pre-population issues.</P> Tue, 04 Feb 2020 19:24:24 GMT https://forums.autodesk.com/t5/arnold-gpu-forum/gpu-cache-population-crashing-with-4-gpus-fine-with-just-1/m-p/11055477#M949 thiago.ize 2020-02-04T19:24:24Z