Pre-Populate GPU Cache

[richardlevene]

Hi,

I am just wondering if there is any difference to using pre-populate gpu cache compared to just pressing render for the first time in the scene with GPU enabled?

You state in the docs in can take 15 mins or more, and it seems to be this sort of time regards how simple the scene is.

Can I ask what it is doing that it needs to take this long?

Thanks,

Richard

[thiago.ize]

Arnold uses NVIDIA Optix for rendering on the GPU and Optix requires us to compile the scene and shaders. This compilation step can take anywhere from a minute to a few minutes at the start of every render. That's a bit painful, so the results of this compilation step are saved into a cache on your disk, so that if you render a new scene with similar settings, fewer things or hopefully nothing at all, needs to be compiled and instead it's all read from disk.

Instead of incurring periodic 1-3 minute pauses when you first start using a new version of Arnold and are changing things in your scene such as adding different lights, toggling volumes, adding curves, meshes, points, etc. you have the option of pre-populating the cache. This will try to compile a whole bunch of different scene configurations and will take roughly 15 minutes. Once it finishes, most scene configurations will have been compiled and saved to disk so that now when you render a previous scene or a new scene, it's likely there will be no compilation delay. Note that we don't cover all combinations at the moment, for instance, even after a pre-population, the first time you add a sphere shape you'll still get a 1-2 minute long pause. But the common cases should be handled.

If you run the pre-population a second time (which you don't need to do), since the cache is already filled, nothing needs to be done and it'll likely take less than a minute to complete (it's not instantaneous because we still need to generate and check that all these different scene configurations are already compiled).

So to summarize, the cache pre-population is not specific to your scene and only needs to be run once when you upgrade your arnold version or (at least at the time of this post) want to render with a new combination of different GPU(s). You don't have to do it, but then you might get occasional "pauses" when you make a change to your scene and Optix needs to compile the new scene configuration. Either way, after using Arnold for a while with the kinds of scenes you care about the cache will eventually get populated one way or another and the slow startup time due to the Optix compiles will go away.

One final note: If it seems like the cache population isn't working, there is a bug in the currently released arnold where using unicode characters in your username will cause the cache to not be properly used. This should hopefully be fixed in our next release.

[richardlevene]

Hi Thiago,

Thanks for the info!

So regardless of your first ever scene you open with Arnold GPU, the pre population will take that time. Even an empty scene.

Where does this info get stored? It doesn’t need to be done again after you close maya, then the workstation off and turn on again with maya and a completely new scene?

Ideally it’s worth running pre-compute gpu cache after you install a new Arnold build, or change your gpu and can be done even with a blank scene?

Thanks,

Richard

[richardlevene]

Also is this something that is just a temporary measure during the early development phase? Is the goal that this will not be needed in the future at all and time to first pixel on very first scene will match the cpu version?

Thanks,

Richard

[thiago.ize]

If you've already been rendering scenes, then the cache pre-population will be faster since some of those combinations will have already been generated. So it can go from 15 minutes, if you start from scratch, all the way down to about 1 minute if you've previously rendered a bunch of different scenes.

On Linux, it's in /var/tmp/OptixCache/

On Windows it's stored in %LOCALAPPDATA%\NVIDIA\OptixCache\%USERNAME%

If you look at the log file and have sufficient verbosity settings enabled, you'll see a message telling you where it's placed. For instance, I get:

00:00:00    55MB         | [gpu] setting OptiX disk cache location to: /var/tmp/OptixCache/thiago/arnold-5.3.0.1_driver-418.56_devices-13330667796282475375

The log file will also have plenty of other useful info such as whether compilation is occurring:

00:01:30  1120MB         |   [gpu] (16/16) Compiled program closest_hit in 1:25.528

This info is stored on disk and stays there forever until you delete it from disk. So even rebooting will leave this around so that future renders can happen quickly.

I think the part that might be causing confusion is what does pre-population have to do with your specific scene. I'll try to give a more detailed example. Suppose you install a new version of Arnold. Then you launch an IPR session of Arnold with an empty scene. When you first add a polymesh object you'll get a 1-2 minute pause. Afterwards, adding or removing polymeshes will happen very quickly. Rebooting your computer and creating a new scene with just polymeshes will also be quick since the polymesh configuration has been saved to the cache on disk. Great! But now you enable volumes and you get another couple minute pause. But after this happens this new configuration is saved to disk and now even after you reboot your machine and create new scenes, you can now enable and disable volumes with polymeshes without any delays. Then you add a curves node to your scene with a volume and polymesh and you get once again the 2 minute slowdown. Then you disable the volume and you get another slowdown because this is a new configuration that was never seen before. Then you remove all the polymeshes so it's just curves and this just curves configuration is also new and so this causes the 1-2 minute slowdown. In other words, if you make a table of all the different configurations of these settings any combination that wasn't seen before will cause this couple minute slowdown the very first time. But once you've compiled it and it's saved to disk, never again will you get a slowdown until you install a new version of arnold.

So all these 1-2 minute pauses can be a little annoying, though in truth they'll only happen a handful of times right after you upgrade arnold and then they'll probably never happen again, even after reboots and new scenes, because it's likely you'll have already populated the cache with these scene configurations.

What if you don't want to deal with these interruptions and would rather just take the 15 minute hit right after you install a new version of Arnold? Then you can optionally PRE-populate the cache. The "pre" means you do it before rendering any of your scenes and this will try to generate all the common configurations so that when you start using arnold you are unlikely to ever see any of these pauses. This is not a step you need to do and in my case I usually don't use it when I upgrade Arnold (I "upgrade" multiple times a day while developing new Arnold features) because I'd rather have the smaller delays come to me one at a time while I use Arnold rather than a big upfront cost. But if I was an end user that only upgraded once in a while, right after installing the new version I'd start the pre-population and go grab some coffee or check my emails.

We definitely would love for the GPU to be as simple to use and as fast to first pixel as the CPU. How or if we are able to match the CPU I can't say. For the time being, I suspect we'll have to keep around the cache and we'll continue to provide the option of pre-populating the cache as a convenience to users that want to get all this work out of the way from the very start.