Make all system paths configurable and overridable using environment variables

[instinctvfx]

In short:

We would like to have all user paths configurable and be able to set them per Max session using environment variables.

Description:

There currently quite some paths that plugins may have to put files into that are not configurable (e.g. maxroot\AMG\.. for viewport shaders). In order to make plugins switchable per sessions these have to be configurable.

In addition the current possibilities for configuring user/system paths leaves a lot to be desired. First and foremost these paths should be lists of paths rather than a single path so there can be multiple icons and scripts paths for example. It should also be possible to override these paths using environment variables to allow per session configuration of plugins to load.

Plugin paths:

I would like to explicitly mention plugin paths. Rather then one (or many) plugin.ini files i would like to have a semicolon separated list of paths that are scanned by max. Along the lines of current env variables the name of the variable could be "ADSK_3DSMAX_x64_2017_PLUGINS".

System paths:

There are also many other paths that plugins need to write to which are configured in "configure system paths". These should also be lists of paths and overridable using environment variables. Proposals for naming them:

ADSK_3DSMAX_x64_2017_ICONS
ADSK_3DSMAX_x64_2017_MACROS
ADSK_3DSMAX_x64_2017_SCRIPTS
ADSK_3DSMAX_x64_2017_STARTUP_SCRIPTS
ADSK_3DSMAX_x64_2017_SHADERS_CACHE
ADSK_3DSMAX_x64_2017_PAGEFILE
ADSK_3DSMAX_x64_2017_TEMP

User paths:

ADSK_3DSMAX_x64_2017_FILE_IO: Allow overriding all File I/O Paths.
ADSK_3DSMAX_x64_2017_EXTERNAL_FILES: Allow adding to the External File Paths
ADSK_3DSMAX_x64_2017_XREFS: Allow adding to the XRefs Paths

Implementation:

I propose to make the environment variables complementary to the current implementation if needed to stay backwards compatible with existing setups. So a default installation would look and behave just like the current setup. IF any of the environment variables are set the corresponding paths would be ADDED to the list of paths for the corresponding asset types.

Rationale:

We are dealing with a lot of different projects that are often very long running and there are typically many in parallel. This means we can hardly find a time to update plugins to new versions. Typically our plugin versions are locked with every Max version. But we neither find a spot to switch Max version easily too for similar reasons.

We are in the process of moving to rez to allow per-session configuration for our tools. This way we could be using different Max+Plugin Versions per project or even more granular. So we could have a VFX fluid shot use some nightly build whereas the rest of the shots would be locked to stable.

Also we could segment our plugins so that only the ones that we need in a specific session are loaded reducing bloat and load times while increasing stability and performance.

[DaveWortley]

Yes, all of the above!!!

[DanielSchmidtBiz]

I think one real main pain point is that plugins should be able to live completely outside of 3ds max. I see that environment variables might help, but also just complicate this whole system even more. I would like to see a real improvement instead of a band aid.

The simplest solution for this mess would maybe be that each plugin directory could contain all the same subdirectories as 3ds max root does for things like icons etc. 3ds max would then also look into all these plugin sub-directories instead of just its own ones.

Instead of plugin.ini files I would like to see the way Photoshop manages plugins. It is very simple but still flexible and organized system.

- there is a central plugin directory

- you can either copy plugins folders directly into it or simply place a link to any outside folder

- subfolders are also searched for plugins (unlike 3ds max plugin directory) so things stay organized and installers don't need to put stuff directly inside central plugin directory

- if you add a ~ to the front of a directory name or name of link this will disable loading of any plugins inside it (makes for easy disabling of plugins or switching between versions)

Daniel

[instinctvfx]

While i agree that a general plugin package definition format would be nice, i was going for a "low hanging fruit" approach as i am requesting this since literally years. That being said i would still prefer the way nuke handles this:

There are multiple standard plugin paths that are always evaluated (like user folder and program folder). In addition you have the NUKE_PATH variable to add as many paths as you can fit in an ENV var. Every plugin in the corresponding folders are loaded and in addition if a menu.py or init.py file are found they are evaluated (both for a GUI session, only init.py for a headless session). This way you can use these two files to do pretty much anything.

There are some problems i am seeing with the approach you described though i may have misunderstood things:

- If each plugin folder could contain all the max subdirectories as the max root does then this would partly contradict parsing subfolders. That would be a total mess and nightmare.

- I'd still need to be able to have an arbitrary amount of plugin folders that i can set via environment variables. With the approach that rez is taking i have all versions of all plugins in the same repository, so i need to be able to not only add all of them, but dedicated individual folders per plugin

- In this context i also would not need the last feature as rez only loads what is configured for a defined context anyways, and as there is no real folder with a set of plugins to load it would not really work at all.

- I can see the purpose of putting links in there, and i can see the advantages, but a single central folder would still not work for us as we aim to be able to load multiple max sessions at once with different combinations of plugins.

Thanks for your input! Keep 'em coming! If we can push this towards a more complete overhaul of the plugin system i am all for it.

Thorsten

[dgorsman]

ENV variables can be a royal PITA to manage (inspect, verify, debug, etc.), more so for casual users who may not have the depth of experience dealing with older OS's where they were more prominent.  They can also be subject to security restrictions.  Working with concatenated strings gets dicey when you have do deal with long strings due to large numbers of deep paths.  They have their place, but IMHO that's for simple "set it and forget it" information rather than per-application-session stuff.

I'm partial to a manifest-driven system, where individual plug-in structures contain a manifest file (probably XML) which dictate dependent folders, files, etc. as required, along with restrictions on program versions they can be loaded in.  Plug-ins with version-dependent settings, content, etc. would existing in separate structures but common content could still be shared.  A top-level, "project" type manifest would dictate which plug-ins are in use, called at start-up through a command-line argument to the EXE.  That allows for start-up under a couple of situations including per-project shortcuts as well as external project manager/launcher type applications.

[instinctvfx]

While i agree on ENV variables having their own share of problems i am not keen on a file based repository system let alone based on XML. What you describe could very well be done using rez packages and i am not keen on wrapping package definition files within package definition files within package definition files (e.g. a python wheel installed as a rez packages that is to be loaded as a max module). Parsing the file system (esp. if it is on a central server) can be quite expensive if many files/folders are involved.

In addition there are two topics mixed that i would really like to be kept separate. Similarly to rez explicitly NOT telling me which packages are to be requested for a certain context (context as in department, project, shot, etc) but based on such a request if the requested combination can be resolved and as a second step give me a shell that has them available.

I definitely would want information about which plugins to load etc. NOT be part of the file system but part of your production tracking system. There may be different departments needing different sets of packages for the same context etc. Finding out which packages and what versions are needed for a specific project by parsing a few hundred shot folders is not really desirable either.

I am definitely open to having different means to configure than environment variables, but i don't think config files are really a better solution.

Another aspect that can be difficult that way is that you can not manipulating PATH can be tricky from running processes, so making max find non-plugin DLLs required by plugins may be tricky if the plugin itself can not set that up properly and has to rely on Max doing so. By wrapping things in rez packages i can set up any additional requirements prior to max starting. Packages have a more active role in such a setup.

If we had a system like nuke where a central init.py could add plugin paths etc. during max startup that would also be a potentially good approach but that still would not solve the problem "which init.py files to load from which folders"

Cheers,

Thorsten

[attilaszabo]

This is a large request.

We can address the problem of plugin link-time dependencies (dlls) having to be installed in the Max root install folder.

Could you guys spell out the high level goal first and agree on that before we decided on what's the best implementation strategy?
Next we could discuss pros and cons of customization techniques. I'm from the C++ school, so the tools I use rely on (a) command line options, (b) env vars, (c) config files. Usually an option can be set via all 3 methods. Sorry, I'm not familiar with "rez" - do you guys have a link to a description?

Thanks

-Attila

[instinctvfx]

Hey Attila and thanks for chiming in!

So essentially my high level goal would be like the following. I may have to revise it a little but these are the main points.

Allow plugins, scripts and other extensions to be loaded from arbitrary locations.

Give us means to dynamically configure these locations per session so that we can run multiple sessions of max with different extensions loaded per session. It is also important that there can be multiple locations so that things like icons do not have to be copied to a single location and plugin packages can stay separate.

From the docs of rez:

Rez is a cross-platform package manager with a difference. Using Rez you can create standalone environments configured for a given set of packages. However, unlike many other package managers, packages are not installed into these standalone environments. Instead, all package versions are installed into a central repository, and standalone environments reference these existing packages. This means that configured environments are lightweight, and very fast to create, often taking just a few seconds to configure despite containing hundreds of packages. 

Here is an overview and the docs:

https://github.com/nerdvegas/rez/wiki

Cheers,

Thorsten

[Anonymous]

High level goal: 

1. Join brand new computer to domain.   
2. Launch 3ds Max into studio environment without any user interaction.   

We can currently join a computer to our domain and it'll launch Nuke with all of our plugins, licenses and everything ready to go.  I can literally launch Nuke within 5 seconds of a fresh Windows installation completing.  Whenever we submit a job to Deadline it'll be rendering on the latest version of Nuke fully configured to render with all the plugins etc.   When I submit a job with 3ds Max... it may or may not render the job properly since it may or may not have any plugins installed or even the correct version of 3ds max. 

[attilaszabo]

Hi im.thatoneguy, thanks for post a high level goal for this request to solve.

What in 3ds Max 2018 is preventing your team from achieving this goal today?

Thanks

Attila

[Anonymous]

• 3ds Max is non-portable.  We can't launch the application from a NAS so it may or may not be on the same patched version. 
• Plugins defined as INI files are difficult to modify and maintain. 
• Many 3ds Max plugins are implemented in a fashion that they spread their files across many directories. 
• Configuring the UI programmatically at startup is nigh on impossible.   Dynamically modifying the CUI files before launch is asking for trouble. 

Solutions:
- Make 3ds Max Batch portable.  Let us launch the 3dsMaxBatch exe from a network share instead of installing to ensure slaves are running a specific build of 3dsmax.exe 

- Let us add a plugin path through Environment variables as part of the render environment for the job. 

- Encourage 3ds Max plugins to be self-contained by making "installation" easy.  With Nuke if you add a folder to NUKE_PATH, you've effectively installed the plugin.  That's all it takes.  Within a NUKE_PATH path you have init.py which configures the software for the plugin and adds the DLLs to the plugin list.  It also contains a menu.py which adds the menus and icons etc.    That way we could keep all of our 3ds Max plugins (including various versions based on the ENV variable plugin path) on a network share.