I think I'm breaking a rule by posting this question here after having posted it in the Mudbox forum, but I hope for a little latitude - it applies in both forums and I think this one is attended by more people, so I think I'm more likely to get an answer here.
I'm using the multi-tile workflow as described in the Area tutorial by Brian Freisinger. The UV's have been laid out using 3D Coat initially, and then optimized and distributed among multiple UV tiles in Maya before sending to Mudbox.
When outputting normal maps with mutlitple tiles, the results are completely unusuable - bright seams, hard edges, a completely corrupt normal map. The process does follow the tiling correctly, but the maps are corrupt and they completely ruin the model.
I've done test after test, and the only thing that fixes this problem is to re-pack these shells into the 0-1 space. With every other option exactly the same, it works with a single tile and breaks with a multi-tile.
Is this something that needs to be fixed in the software, or am I missing something? I've looked through Brian's tutorial and watched the videos on Digital Tutors and there doesn't seem to be anything different that needs to be done for these maps to export correctly. As I said - with everything else being equal and this being the only variable, it works fine.
Can someone please address whether or not this is something broken in the software or with my workflow?
Thanks
Solved! Go to Solution.
Solved by _sebastian_f. Go to Solution.
After much experimentation and ultimately a conversation with Brian (the author of the tutorial that appears on this site) we narrowed down the source of the problem and (Brian) came up with a workaround.
After following Brian's tutorial, normal mapping using multiple tiles fails. The source of the problem is the Maya "Layered Texture" node, which works fine for painted textures but not at all for normal maps. The normal maps produced with Mudbox for multiple tiles work fine; we originally thought they were the issue.
Although Brian's tutorial doesn't specifically address normal maps, the same concept is described in a Digital Tutors tutorial that does. Further, in response to this same question being posted in the Mudbox forum, Tony Su from Autodesk demonstrated that he had no problem with this same setup. Perhaps this issue arises with more than two layers in the layered texture node.
We followed this tutorial in a class at a university, and it failed on 20 lab computers and a number of student-owned machines. It failed with SP4 and SP5. It failed with a clean boot and with default preferences.
The workaround, which probably ought to be the recommended workflow anyway, is to eliminate the "Layered Texture" node altogether in the shader tree - and sequentially connect each normal map to the "Default Color" of the one before it. Thus, the map designated for the 1-2 texture space would be connected to the default color of the one designated for the 0-1 space; the 2-3 map would be connected to the default color of the one designated for the 1-2 space, etc. Only the original normal map (0-1) was connected to the bump, which was set to "Tangent Space Normals" and rendered in mental ray.
The results are perfect.