MentalRay bug: Incorrect tangent normal map calculations
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report
Hello,
During our shader programming, we have spent weeks trying to understand why MentalRay sends us degenerated results when calculating normal maps.
At first we have thought that our work was based on wrong calculations, until someone here came to the idea to test 3dsMax with a generic sphere and a simple tangent normal map. The result is quite disturbing. It is hard to explain what is happening, but if this is the way how MentalRay renders, than it is very certain our all images since 2012 are rendered with incorrect normal maps calculation.
After double and triple checking all parameters, SDK, google and other software, we can only hope somebody here will shout "hey, press the x check-box then it works correct!".
What is wrong with MentalRay tangent normal map calculation?
Any user can very simply verify the correctness of the rendering. It is so obvious that almost no explanation is needed.
Step 1:
Create a simple sphere, and light above it (or wherever).
Assign Arch&Design or Standard Shader to it and add into Bump/Normal a Tangent Space normal texture (things you do million times a day)

simple scene setup
Step 2.
Duplicate the object. Now, move it in the scene away, and translate it (Rotate around, move around, important is that the mesh has change the world space position and orientation). DO NOT Mirror it, just move and rotate it in the scene, away from 0,0,0.

Copy and translate/rotate object
Step 3.
Render the scene with [U]MentalRay[/U].
What you see is at the first glance a good image, until you look exactly at what has happened.
On the copied and translated mesh, MetalRay has altered the Normals. The Tangent are not correct anymore, the "cap" is on the original image "extruded, elevated", while the same area on the rotated object is pressed inside, dented. This is actually wrong - after rotating object the "hills" should elevate outwards. Observe the copied mesh and look at this kind of "negative" tangent space values (or whatever MentalRay has calculated). This may look like a nice relief, yet after short thinking you will notice an absurd error.

Copy and translate/rotate object
Step 4.
Change the renderer from MentalRay to Vray. Render the image and examine the results.
You will prompt see that Vray has rendered the normal maps in correct way. The copied mesh does not have inverted tangent normals.

MetalRay - Vray comparison
Step 5.
I have re-rendered the same scene with all 4 Max Render engines i have:
- MentalRay
- Vray
- FinalRender
- Scanline
All render engines save MentalRay have rendered the normal map on the translated object in correct way.
(yes, i have tested with different gammas, resetXFroms, materials, lights, etc; it does not affect the incorrect output)

MetalRay - Vray - FinalRender - Scanline comparison
This is very weird!
At the end, i have done the same thing in Maya 2016 Ext2 and rendered the scene with Maya MentalRay. And the things here start to get very strange: Maya does suddenly the same error.

MetalRay Maya 2016 Ext2
This is almost paradoxical: we have been working since 2012 with wrong tangent map representation (I do not have Max2011 anymore, so i can't check if it goes deeper).
Either no one cares about how his normal maps look like, or no one uses MentalRay at all. I hope the second is true.
Even if all 3d artists do not care how their NormalMaps look like on different randomized objects in the scene, what about game-designers? Did you notice that render-to-texture (texture baking) with MentalRay delivers flipped, twisted and wrong Tangent Normal Maps, which make your realtime models look strange and wrong? All this is based on the core anomaly you have seen above.
There is still a hope some technical guys here will point me at some fundamental logical error, otherwise you can trash the whole renderer.
thanks for your inputs in advance
