Normal Map Baking Frustration

[Anonymous]

Hello all!

For today and most of yesterday, I could not figure out why my normal maps have baked out so weirdly (photo references below). I am a student that has learned everything thus far on my own, but I've never hit a rock this hard before. No matter what options I seem to alter, what program I use to 'transfer'/'generate' maps, what direction vertices are facing, and what other methods I experiment with, I can't get my normal maps to work correctly. I've tried to map a pole of a lamppost, the lantern of a lamppost, and 2 different kinds of rocks. They all look similar to the mapped picture below in that some lines appear random and out of place.

I have tried doing the following, but they all gave me a rubbish ugly normal map that, when rendered, makes no sense to the eye whatsoever:

• averaged vertice normals
• set vertice normals to face
• smoothed edges
• hardened edges
• modelled the envolope/cage as close as I could to the high poly mesh
• experimented with many of the baking options
• sewed many UV edges together. I did not notice a change in the normal map
• 'automatic mapped' a UV layout and baked it
• exported as obj to xNormal
• exported as fbx to xNormal.. just for the heck of it

If someone could help me out that would amazing, or if someone could answer one or two of these questions, that'd also be super!

1. Can certain UV placement or sewing certain UV edges improve normal maps? Do UV's impact the baking outcome at all?
2. Does the high poly mesh need the exact UV layout as the low poly?
3. Will vertice normals impact normal maps? (though I assume that Maya automatically averages the normals for the sake of baking?)
4. What are the 2 best file formats to export into xNormal?
5. Is there anything wrong with my models that is making this problem?
6. *BONUS* do you enjoy the new 2016 hypershade? It's a real hoot and a holla' for us newbies.  

Applied normal map to low poly:

low poly:

high poly:

[_sebastian_f · Accepted solution]

1.  yes uv maps play a big role when normal mapping. you need non-overlapping uv´s first of all. even when you can bake objects where the uv´s are mirrored or stacked you´ll need to move the mirrored/stacked uv´s out of the 0-1 space for baking. there are exceptions for the following but as rule of thumb: where you have extreme angles between faces you´ll need most likely hard edges for good shading and therefore uv splits or more geometry/bevels for a good normal map. it´s not necessary the other way around but it´s often the case that you place seams where you have hard edges when doing hardsurface work so there are scripts out there hardening your edges where you have uv seams. 

2. the highpoly doesn´t need uv´s at all.

3. face/vertex normals matter (see above). fortunately maya doesn´t do anything to the normals on it´s own otherwise you get into trouble.

4. fbx should be fine for xn but it has also it´s own file format which you can use to export from maya. while xn is a nice tool i would start with getting a good looking normal map in maya first. it´s slower but the quality is good. in xnormal you have to start to think about cage creation as well.

5. from the screenshots the models look fine. i guess its a uv and/or normal problem.

6. don´t have problems with the new hypershade: if you have specific questions let me know? what´s the difference for a "newbie"?

last but not least read this: http://polycount.com/discussion/107196/youre-making-me-hard-making-sense-of-hard-edges-uvs-normal-ma...

hope it helps.

[Anonymous]

"hope it help"

I think you've covered everything I need to know and posted (what looks like) a great article/tutorial for me to read. Thank you so much - A+ response!

The new Hypershade is a little frustrating for us newbies when we get stuck and would maybe like a tutorial to watch or read about, which are non-existant for the new Hypershade. There's also an annoying fatal bug where if you delete a material in the work area, sometimes it doesn't delete from the selection area and if you reselect the deleted material then Maya crashes. I still haven't found a glass preset material which I think gets me the most, but I sympathize for any new students doing what I'm doing (learning everything on their own) because they changed a few terms around, and of course, it looks very different. And once students get too overwelmed, it's very easy to lose interest, lose focus, and give up; I think the new hypershade can easily overwelm new students who learn on their own.

[_sebastian_f]

you´re welcome! feel free to return when you have more questions.

as for the hypershade: i thought not to much changed so old tutorials should still apply. one problem maybe that by default you don´t see objects anymore? you can rmb in the work area and go to show > objects and check what you like to see. for glass presets the mia material offers some. "problem" with materials like glass, they highly depend on the environment to look good.

"...sometimes it doesn't delete from the selection area and if you reselect the deleted material then Maya crashes. "
this sounds like a serious problem. never experienced this so i have no idea. if you find a way to reproduce it: report it! one thing you can always try: make sure you are on the latest version and/or backup your prefs, reset them and see if the problem persists.