GenerateNormals() completely ignores hard edges for me, softening and distorting the normals, even when smoothing groups is turned on for the export.
This is not a solution.
@ilan.keshet is right in his post and it has probably been broken in exported files, and with no workaround it's just weird, inconsistent behavior, period. It's a mistake, a coding error. You haven't even expressed a valid reason why this behavior WOULD exist, otherwise why isn't it in 3ds Max? It isn't in 3ds Max because a different development team made it long ago and didn't make the mistake.
In fact I've discovered another bug in Maya's export, or perhaps a "by design" "bad design" limitation. If I attach a mesh to a bone, all hard edges are ignored on export. Is that "by design"? Or do I file a report on that also?
Dave
@regalir wrote:
And while, at first, it seems obvious that the exported normals should not be dependent on the skin deformation at a given time, I am not sure that this assumption still hold true if the deformation is actually a combination of skins, blend shapes and/or vertex caches and by the way each of these systems interact with each other. I guess that in my previous posts I created more confusion than provided answers. I apologize for that!
It seems obvious because it's correct. The assumption does hold true, end of story. You can't defend the indefensible and no amount of word salad can change it. (All those systems interacting! You mean the systems I'm also doing in my own engine? And so on.)
But this is a Maya bug, and not the FBX SDK at all obviously. Your team has been winning for a long time, otherwise support for FBX wouldn't be critical in every 3D package and engine.
What's likely is someone from Alias in 2002 or something had a late night out partying on LotR money and wrote the data out from the deformed normals array and not the mesh, clocked out and called it a day.
When Alias wrote the OBJ export, there was no skinning, so they would have used the deformed normals, because had to use the deformed geometry. When it came time to write the FBX export, they perhaps copied code and used the deformed normals by mistake.
It was never corrected because it's probably rarely encountered. Most skinned meshes are going to be put into bind pose for export, and a lot of times animations are placed in separate files. But my example is just a simple animated mesh in one file, and it breaks.
Having calculated deformations for one pose on import is totally useless to 100% of people importing from FBX, so Maya should make a correction. Otherwise, why not deform the geometry too? Because it doesn't make sense. You'd have to unwind all deformations to get the geometry, what a pain! Just like you have to unwind all deformations to get the normals. Doesn't make sense.
I'm concerned that the solution proposed calculates the smoothing based on deformed normals. It's probably fine, but I can see theoretical issues where normals could be smoothed when they weren't supposed to be. Hmm. I'm surprised FBX doesn't know about hard/soft edges and they have to be computed from normals in the first place. Such dirty workarounds for one mistake!
And thanks for your help. This workaround is probably satisfactory, but I'll need to try it. I know you are trying to solve an issue for us rather than take a challenge to the Maya team. They wouldn't be able to defend it but they have a big ship that moves slowly and they probably wouldn't prioritize a fix even if they acknowledge it. But I would be happy to still report it to them and it would be interesting to see their answer.
Let me put the cherry on top for you and for @ilan.keshet also. Here's Blender importing a Maya mesh (not using FBX SDK) and the normals are wrong in the exact same way as in my engine:
In the above example, you can see that Blender then deforms the mesh into the pose, meaning the normals are now double-transformed (creating the very dark wing). Blender's devs didn't catch the issue; @ilan.keshet and I did.
Blender offers the option to not accept the normals, in which case Blender re-computes the normals:
...and hard edges are lost... ooooof!
So they are trusting the export purely as FBX describes it. They may be unaware of the Maya export issue, but it would require something like what we're discussing to work around it, at least partly.
But no workaround appears to solve the matter. All of this can be avoided if the normals were not baked, and it would be nice to see future versions of Maya not have this problem. All our suffering would end!
Dave
I have managed to solve the problem finally... after much discussion + testing.
In recap, the problem exists strictly in Maya's FBX Export. -- Maya's raw normals attempts to keep locked normals and such -- and works great when it comes to static geometry, -- but exports deformed normals if mesh is skinned.
The solution is to be able to Generate the normals based upon smoothing information in the mesh.
The posed solution that reglir has given, -- is the right idea, -- except the fact that the GenerateNormals() function provided by FBX has a critical bug in it -- It works in most cases, but in the case of Dave's model he was showing fails miserably.
The case where the FbxMesh GenerateNormals fails can be described in the image below, where the red line is a hard edge, and the rest are all soft edges.
In this case, there are 3 polygons above that should be smoothed together, -- and 3 polygons below that should be smoothed together. Unfortunately, -- in this case the FbxMesh GenerateNormals function fails, and instead shares 1 normal with all 6 of the triangles.
How did I go about figuring this out?
Well, I decided to rewrite the entire GenerateNormals myself using the smoothing information, -- and managed to produce the exact same bug that FbxMesh's GenerateNormals does --
After figuring out this bug, then I handled this case in code, and it fixed everything that Dave was mentioning.
Results:
Using Fbx's GenerateNormals (and also I produced this before handling the case listed above)
Using my GenerateNormals:
As you can see the hard edge in the wing is missing in the top picture.
I used https://knowledge.autodesk.com/support/maya/learn-explore/caas/sfdcarticles/sfdcarticles/How-Maya-co... as a reference on how to write the code -- though this reference in itself has bugs in it.
Here is the algorithm that I used that did work:
SharedEdge //This contains all vertexIds that are on the an Edge
{
startControlPointIndex = -1
startVertexId = -1
startVertexId2 = -1
endControlPointIndex = -1
endVertexId = -1
endVertexId2 = -1
};
Clear Normals IndexArray / DirectArray
normals->SetMappingMode(FbxGeometryElement::eByPolygonVertex);
normals->SetReferenceMode(FbxGeometryElement::eIndexToDirect);
map<int, SharedEdge> smoothEdges{};
Loop over all polygon vertices and build the smoothEdges map with key being edgeIndex, and a SmoothedEdge
edgeIndex can be found using: mesh->GetMeshEdgeIndexForPolygon
Check if edge is smooth using SmoothingGroups
Loop over all edges in SharedEdge map
Group startVertexId + startVertexId2 in same shared normal grouping
Group endVertexId + endVertexId2 in same shared normal grouping
Loop over all groupings found
Add a Normal into the Direct Array + link to this index from the group
Loop over all polygons
if (vertexId is a sharedSmoothNormal)
compute: normals[normalIndex] += faceNormal * vertexAngle * faceArea;
else
Add a shared normal into the direct array (Hard Normal)
normals[normalIndex] = faceNormal
Finally, and as a last step, ensure all smoothed shared normals have been properly normalized.
@regalir, I can confirm @ilan.keshet 's post. He was able to recalculate normals properly when GenerateNormals() would not, while following your procedure. The procedure would work, but GenerateNormals() does not work. For my mesh, it completely smooths every vertex, even though there is correct edge-based smoothing data in the FBX (and converting to polygon smoothing doesn't work either).
I followed essentially the same algorithm with different structures and had success.
So, the correct answer is... write it yourself. Then, why doesn't GenerateNormals() work as described?
Dave
