topic Re: MentalRay bug: Incorrect tangent normal map calculations in 3ds Max Forum

MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Sat, 07 May 2016 01:33:06 GMT

Hello,

During our shader programming, we have spent weeks trying to understand why MentalRay sends us degenerated results when calculating normal maps.

At first we have thought that our work was based on wrong calculations, until someone here came to the idea to test 3dsMax with a generic sphere and a simple tangent normal map. The result is quite disturbing. It is hard to explain what is happening, but if this is the way how MentalRay renders, than it is very certain our all images since 2012 are rendered with incorrect normal maps calculation.

After double and triple checking all parameters, SDK, google and other software, we can only hope somebody here will shout "hey, press the x check-box then it works correct!".

What is wrong with MentalRay tangent normal map calculation?

Any user can very simply verify the correctness of the rendering. It is so obvious that almost no explanation is needed.

Step 1:

Create a simple sphere, and light above it (or wherever).

Assign Arch&Design or Standard Shader to it and add into Bump/Normal a Tangent Space normal texture (things you do million times a day)

simple scene setup

Step 2.

Duplicate the object. Now, move it in the scene away, and translate it (Rotate around, move around, important is that the mesh has change the world space position and orientation). DO NOT Mirror it, just move and rotate it in the scene, away from 0,0,0.

Copy and translate/rotate object

Step 3.

Render the scene with [U]MentalRay[/U].

What you see is at the first glance a good image, until you look exactly at what has happened.

On the copied and translated mesh, MetalRay has altered the Normals. The Tangent are not correct anymore, the "cap" is on the original image "extruded, elevated", while the same area on the rotated object is pressed inside, dented. This is actually wrong - after rotating object the "hills" should elevate outwards. Observe the copied mesh and look at this kind of "negative" tangent space values (or whatever MentalRay has calculated). This may look like a nice relief, yet after short thinking you will notice an absurd error.

Copy and translate/rotate object

Step 4.

Change the renderer from MentalRay to Vray. Render the image and examine the results.

You will prompt see that Vray has rendered the normal maps in correct way. The copied mesh does not have inverted tangent normals.

MetalRay - Vray comparison

Step 5.

I have re-rendered the same scene with all 4 Max Render engines i have:

- MentalRay

- Vray

- FinalRender

- Scanline

All render engines save MentalRay have rendered the normal map on the translated object in correct way.

(yes, i have tested with different gammas, resetXFroms, materials, lights, etc; it does not affect the incorrect output)

MetalRay - Vray - FinalRender - Scanline comparison

This is very weird!

At the end, i have done the same thing in Maya 2016 Ext2 and rendered the scene with Maya MentalRay. And the things here start to get very strange: Maya does suddenly the same error.

MetalRay Maya 2016 Ext2

This is almost paradoxical: we have been working since 2012 with wrong tangent map representation (I do not have Max2011 anymore, so i can't check if it goes deeper).

Either no one cares about how his normal maps look like, or no one uses MentalRay at all. I hope the second is true.

Even if all 3d artists do not care how their NormalMaps look like on different randomized objects in the scene, what about game-designers? Did you notice that render-to-texture (texture baking) with MentalRay delivers flipped, twisted and wrong Tangent Normal Maps, which make your realtime models look strange and wrong? All this is based on the core anomaly you have seen above.

There is still a hope some technical guys here will point me at some fundamental logical error, otherwise you can trash the whole renderer.

thanks for your inputs in advance

Re: MentalRay bug: Incorrect tangent normal map calculations

senorpablo — Sat, 07 May 2016 03:55:16 GMT

Interesting. I can't say I'd be surprised if what you seem to have illustrated is true.

I know that in Iray, for example, the green channel needed to be flipped, meaning it wasn't consistant with the scanline renderer. I believe Iray is based in part of Mental Ray.

One thing you might check is also to make sure you have gamma override set to 1 for your normal maps. Auto gamma will give bad results. What you are showing doesn't look like it's a gamma issue, however.

Re: MentalRay bug: Incorrect tangent normal map calculations

RGhost77 — Sat, 07 May 2016 05:05:39 GMT

I think this can be posted also at https://forum.nvidia-arc.com, maybe guys from nvidia have the answers...

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Sat, 07 May 2016 15:49:53 GMT

hello Seniorpablo,

thank you for the feedback.

IRay renders correct, this is MentalRay anomaly.

As written in the text, Gamma, FlippedYX are not the subject: all combination has been tested and deliver the same error. Both spheres you see in the image are rendered together, with the difererence that the left one was moved and rotated. Changing Y or Gamma would mean i change automatically the look of the right sphere, since they share the same material.

And this all does not explain why Vray, FinalRender, Scanline, Maxwell, Blender etc does the calculation in correct way.

I assume no one has noticed this so far. On the other hand - how can you not notice that the folds and bumps on the characters face got suddenly inverted if it changes the position and rotation....

thanks for the input!

best regards

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Tue, 07 Mar 2017 11:15:32 GMT

Same problem here. Mental Ray SSS shader. The right one is the wrong the left one is the good, you can see the back line.

Only difference are:

right one: the head rotate

left one: only the light and camera rotate around the head

So looks when I rotate the model the normal vectors not follow this....

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Tue, 07 Mar 2017 11:37:30 GMT

Looks mental ray change wrong tangent when model rotate.

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Tue, 07 Mar 2017 15:22:31 GMT

Hi Bonbonpuli,

at last someone noticed it. This should be actually the sticky issue in the max Forum, since it is a Major bug preventing you to render correct images. I wonder why no one has noticed so far that all game characters, architecture and industry stuff rendering outputs are totaly defect.

But now, since MentalRay is dead, i don't think it will be ever fixed.

RIP

best regards

Re: MentalRay bug: Incorrect tangent normal map calculations

Alfred.DeFlaminis — Thu, 16 Mar 2017 00:39:51 GMT

Hello @Anonymous,

This thread is before my time here but you've done a great job documenting all of this. You would think that people would notice the problem in viz and commercials, but gamers wouldn't care about MR display of normal maps. Unless you mean a studio rendering animations for cutscenes or intro movies. I don't think MR is dead at all though, it just won't be integrated with Max.

You might post this on the NVIDIA forums, I'm sure they'd be interested in the issue and make a move to fix it. In the meanwhile I will go ahead and add this to the tracker on my end so that we've covered both bases. Thanks for reporting this.

Best Regards,

Re: MentalRay bug: Incorrect tangent normal map calculations

Alfred.DeFlaminis — Thu, 16 Mar 2017 23:50:55 GMT

Hello @Anonymous,

I just wanted to follow up here and mention that NVIDIA is forging ahead with Mental Ray in Max and have plans for some great new features. So I don't think MR is dead at all, it's just moving to NVIDIA. I think this is great news all around for everyone who uses MR and I have more information here.

I hope this puts to rest the idea that MR is dead, it isn't. Thanks!

Best Regards,

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Mon, 20 Mar 2017 22:19:51 GMT

Hello Alfred,

thank you for your support and the new informations, you are very helpfull!

When i said "MentalRay is dead" I did not mean the development is dead. I was rather pointing at the fact that, according to our own plugin sales for 3dsmax, at least 60% of all users work in architecture and product visualisation. They must have noticed that their walls and streets, their cars and vacuum cleaners react completly wrong to the light and reflection in the advertising shots. Or at least the customers would have noticed the quality can not compete with high-end renderings from the competitors.

Therefore was my conclusion "MentalRay is dead" -> for the proffessional usage: it seems they all switched to Vray and no one will ever notice this anomaly since no one uses it anymore.

I am looking forward to see the MR Release for Max 2018 and if it got some good progress (we still use MR is some productions, you have 999 render nodes which is a huge production advantage) 🙂

thanks

best regards

Igor Posavec

www.3d-plugin.com

Re: MentalRay bug: Incorrect tangent normal map calculations

Alfred.DeFlaminis — Mon, 20 Mar 2017 22:38:48 GMT

Thank you for the clarification and update @Anonymous, I really appreciate it. I understand what you're saying and I do see a lot of Vray lately. I'm very excited though that NVIDIA is moving forward with MR. Competition only makes things better for us on the user level and NVIDIA has been so important over the past decades that I'd hate to see MR go away. Thanks for much for taking the time to share you thoughts with me!

Best Regards,

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Mon, 22 May 2017 11:40:32 GMT

hello,

just to update the thread:

NOTHING has been repaired in the latest Mental Ray 2018. This bug still persists, you can't effectivly use MR for any kind of serious production.

best regards

Re: MentalRay bug: Incorrect tangent normal map calculations

Alfred.DeFlaminis — Mon, 22 May 2017 16:48:16 GMT

Hello @Anonymous,

For 2018 and above any problems with MR should be reported on the NVIDIA forums if possible. I actually found your thread there so I know you've done this, but I wanted to mention it for any others that find this thread. Development of MR for Max 2018+ is being handled entirely by NVIDIA. Thanks!

Best Regards,

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Wed, 24 May 2017 16:18:45 GMT

Hello Alfred

thank you for your fast response - i have posted it in both Forums, in order that all max and mr users get informed about the Progress. Once it is fixed i will post it here too 🙂

thanks for support!
best regards

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Thu, 16 Nov 2017 23:37:30 GMT

Hi.

Have you tried to flip the Y direction of the Normal map?

I've read somewhere that some programs have better output results when the Y vector is flipped upon the generation of the normal map.

(In Zbrush for example.)

Maybe it works better for Maya.... but I could be entirely wrong..

Re: MentalRay bug: Incorrect tangent normal map calculations

Anonymous — Thu, 16 Nov 2017 23:43:16 GMT

no, the bug is confirmed by Nvidia. So far a year or more have passed, with the current speed i do not think we will get the fix in the near future.

In the meantime I have had to abandon MR, unfortunately 😞

thanks

best regards

Re: MentalRay bug: Incorrect tangent normal map calculations

Alfred.DeFlaminis — Fri, 17 Nov 2017 00:08:23 GMT

I wonder if the normal bump mode in the Preferences would have any effect.