PROBLEM:
(see the image below) Two simplest passes where the pass that is plugged into the technique as second should overwrite the first, but it produces an error.
REPRODUCTION:
See the image below for reproduction. (note that the error state might look different on your hw or settings if you're trying to reproduce. use dx11, for me in dx11 viewport2.0 mode shaderfx error state makes the object turn green and error log prints in script editor (gotta scroll up abit). in opengl mode, the object goes white and maya doesnt show a red error status line message but the shader doesnt compute (can verify by setting the pixelshader inputs to dark, object will still be white)
the graph in the image should error out BUT if you disconnect one of the pixel shader color inputs while in dx11 viewport mode, the pixelshader gets auto assigned black and then the graph works and we can see the color of the second pass properly update on change.
FURTHER DETAILS / OBSERVATIONS:
after investigating the errors (all done in dx11 viewport mode, where the errors show up in script editor or can export the shader to disk from the hwshader node property window) it turns out to be related to some hardware fog code that shaderfx autoinserts into the code and that produces few different errors based on the graph setup:
error1 - if i use a shared vertexshader for both passes and separate pixelshaders with color inputs, it inserts that hardware fog code into the shared vertexshader twice (because each pixelshader requires the results) = error: "_HPosition var redeclaration". but if one color input is disconnected from a pixelshader, it doenst produce that code inside the shared vertexshader twice because the second PS doesnt need that data, so it works. (at least in dx11, i think in opengl it doesnt get the default black autoasigned and errors out)
error2 - if i use separate vertexshaders for each pass and separate pixelshaders with color inputs, it puts the fog code in each VS properly but then a different problem with undeclared/missing structs/vars comes: "WorldPosition var undeclared"
TRIED:
also tried reconstructing the vertex shader correctly from scratch (shaderfx.com tutorial) and using separate one for each pass and renaming vars and texcoords so nothing collides but then few more different problems occured (problems with vs struct declarations/naming in the resulting code) which would be offtopic here. either way, each pass always worked on its own but never when plugged in together and having a separate pixel shader with color inputs, i never managed to find a combination of graph setup and viewport settings that would make this simple scenario actually compute once. if i remove the fog code in the exported hlsl and run it through mayas dx11shader it works as expected.
Graph to reproduce error:
been stuck on this for days now, any kind of input is appreciated if you have a moment to reproduce this please let me know what happens on your side or whether im doing something terribly wrong even in such a simple graph ><.
(maya 2018/2017 (tried default preffs), dx11 viewport mode (but wont work in opengl too), win 8.1 64, ati HD7950 Flex 3gb)