Run an expression when switching to a render layer

[DavidHenion]

Have a scene with some boolean objects for which I want to change the bool operation on a render layer.

I've tried just doing a normal override, but something like polyBoolean1.operation is unsupported, but I can write a script to find them and change the operation - just not sure how to get it to run for a render layer.

Currently using 2023.1

[Kahylan · Accepted solution]

Hi!

So I gave your problem some thought and I actually came up with a solution that I think should work. I can't really be sure, since I can't test it. But the tests I could do in Maya 2022 looked promising.

There are very few types of Default Nodes that can execute code (scriptNodes and Expressions), those types only listens to certain triggers and switching renderlayers sadly isn't one of them. So I think using an expression for this acutally isn't the right approach.

The problem you have is that int32Array attributes are not compadible with the layer override system, so you basically need a proxy value that changes the .operation attribute. You can achieve this by using a choice node and intArray Attributes, like in the setup pictured below:

the choice node can take attributes of any type and switch between them as long as they are the same type, so it works with Int32Array attributes. All you need to do is create two Int32Array attributes on another node/nodes (both can be on the same node or different nodes, just not the node that the choice feeds into) and set those attributes to the Arrays that you need to output in the end. Then you connect them to the choice node like pictured and connect the output of the choice node to your "polyBoolean.operation". Use the layer override on the choice nodes "selector" attribute.

Int32Array attributes can only be added/edited via code, here is an example of how to do that in python:

import maya.cmds as mc
mc.addAttr("Name_Of_The_Node", ln = "Name_of_Array", dt = "Int32Array")
mc.setAttr("Name_Of_The_Node.Name_of_Array", (1, 1, 2, 4), type = "Int32Array")

I know you asked for MEL synthax, but for some reason MEL wouldn't let me set an Arrayattribute with more than 3 intergers and MEL isn't my main language so I figured I just give you the python synthax, you might be better at MEL than I am and able to figure that problem out if your really need MEL. 🙂

I hope it helps!

[DavidHenion]

Awesome - thank you this works! With one modification...

No matter what I tried I couldn't get the choice1.selector attribute to be affected by the render layer absOverride node. I put the choice node in its own collection, added the overrride, typed in the attribute, set its value - but it just never seemed to affect the actual node.

I put my arrays on an empty transform node similar to your example - so I thought why not use something simple, like that node's transforms, to drive the selector - so connected ArrayHolder.tx to choice1.selector, added ArrayHolder to a collection in the render layer, added an override for the transform, and set the transform override to 1,0,0 and bam, it works!

(choice1.selector as 0 selects the ArrayHolder.SetDiff array, choice1.selector as 1 selects the ArrayHolder.SetIntersect array.)

Screenshot of this attached.

Two side notes about arrays-

Here is the syntax for adding and setting in MEL:

select someNode;
addAttr -ln newAttributeName -dt Int32Array;
setAttr someNode.newAttributeName -type Int32Array 3 2 1 0;
getAttr someNode.newAttributeName;
// Result: 2 1 0

Also, I was totally wrong about thinking the bool operation was a 4 element array - it is 2, but the setAttr command seems to be kinda weird about setting arrays.
Every reference I can find to defining an array in MEL is done by just giving an array its values. But the setAttr command takes the first number as the number of elements in the array.

BUT - setAttr returns some very odd messages if the count is off:

setAttr someNode.newAttributeName -type Int32Array 4 1 0;
// Error: line 1: setAttr: Error reading data element number 6:

element number 6???

setAttr someNode.newAttributeName -type Int32Array 2 1 1 1 1 1;
// Error: line 1: setAttr: Too much data was provided. The last 2 elements were not used.

only the last 2???

So my arrays on the ArrayHolder node are set as follows:

getAttr ArrayHolder.SetDiff;
// Result: 2 2
getAttr ArrayHolder.SetIntersect;
// Result: 2 3

I don't fully understand how the two values are used by the boolean's operation value - but these values work for me.

Thank you again for all your help!

[Kahylan]

Great that it works!

It's weird that you couldn't connect the choice nodes selector attribute directly, because I tried it out in Maya 2022 and it worked there and I couldn't find anything in the docs about changes to the choice node it 2023. The selector attribute is just a simple integer attribute which the absOverride should accept.

Did you have your Collections "Collection Filters" setting set to "All"?

Thanks for the info on the MEL setAttr, I so rarely encounter int Array attributes that I actually did not know that^^

[DavidHenion]

Argh - I've never used collection filters - and yes it was set to 'transforms'. I was able to hook it up though - it all was there and all looked good, just with the filters not set to all it just didn't do anything. I figured filters was more for creating connections - not turning off ones that are already connected.

With the setup in the attached screenshot - if I change the collection filter to 'all' - it works.

Ah well, lesson learned - thank you again for all your help!