I have an FBX where normal, tangent and UVs layers are created with mapping mode eByPolygonVertex and reference mode eIndexToDirect.
After I edit the FBX in Maya 2016.5 and export it, the reference mode changed.
question number 1: Why Maya changes the reference mode in layers?
In order for my tools to work correctly with the FBX file I need to change the reference mode to eIndexToDirect, for example in the case of the Normal layer this is what I am doing:
FbxGeometryElementNormal* lGeometryElementNormal = pMesh->GetElementNormal();
if (lGeometryElementNormal->GetReferenceMode() != FbxGeometryElement::eIndexToDirect)
{
//FBX mesh with unsupported reference mode
const fbxsdk::FbxLayerContainer* pContainer = lGeometryElementNormal->GetOwner();
pContainer->ConvertDirectToIndexToDirect(0);
}
but this doesn't even compile, the error I get is:
the object has type qualifiers that are not compatible with the member function "fbxsdk::FbxLayerContainer::ConvertDirectToIndexToDirect"
so ...question number 2: What I am doing wrong, and what is the right way to convert the layers reference mode?
thanks,
Emanuel