Importing iPhone X Face Mocap data into Maya.

[harshad.asawale]

I was pretty excited since I came across this video.  

The face Mocap looks pretty solid and looks like something which can save countless hours of animations.

So basically how the whole thing works is

- Face mocap data is captured using unity app and saved in a text file.

- This data consists of values for 51 face blendshapes and transform of the head origin.

- Then it is formatted into a .anim file, which can be imported into Maya.

You can read the whole article here.

I was able to get the whole mocap data into a json (file)

But currently stuck at the .anim file exporter.  

There are a few parameters (even though they are explained here) which are currently not making any sense to me as I am not an animator. 

If anyone can help me out with understanding the following piece of data from .anim file, it will be very helpful.

Thanks

anim translate.translateY 0 2 1;
animData {
  input time;
  output linear;
  weighted 0;
  preInfinity constant;
  postInfinity constant;
  keys {
    0 0 auto auto 1 0 0;
    1 0 auto fixed 1 0 0 0 1;
    20 0.24418778 fixed fixed 1 0 0 0.74613416 1 0.74613416 1;
    35 0.47890941 fixed fixed 1 0 0 0.61673066 1 0.61673066 1;
  }
}

[chaneyx · Accepted solution]

You will need to experiment with exporting and reimporting animation to understand these things.
Alternatively, you can write your own MEL or Python imported which grabs the per frame keyframe 
values you need and setKeyframes in the standard way - ie. setKeyframe -at scale;

But here is some partial info off for you : 
_________________________________________________________________
This is required by the MAnimCurveClipboard. 
anim translate.translateY 0 2 1;
It usually looks something like this : 
anim translate.translateY translateY pSphere1 0 1 1;
Essentially it allows for curves to be pasted into hierarchies. 
The 0 2 1 is the slightly cryptic part as it is specific to MAnimCurveClipboard.
animData {
In your case, this will always be "time" as your animation is time driven  input time;
Output keywords refer to the type of unit it is. 
Rotations(angular), distances(linear), scales (unitless) are the most common.
Be aware that you may need to do a unit conversion with these values when dealing with rotation
  output linear;
This is all the curve information
A basic read on the Windows->Animation->Graph Editor will help
For example, postInfinity refers to the extrapolation after the last keyframe.   weighted 0;
  preInfinity constant;
  postInfinity constant;
This is the data you really want. It is your actual keyframes! 
You will need to understand the Graph Editor tangents. 
And you should look at the animTL and animTA curves in the hypergraph to understand
20 - frame 20
0.24418778 - value at frame 20
fixed - in tangent type
fixed - out tangent type
1 0 0 0.74613416 1 0.74613416 1  keys {
    0 0 auto auto 1 0 0;
    1 0 auto fixed 1 0 0 0 1;
    20 0.24418778 fixed fixed 1 0 0 0.74613416 1 0.74613416 1;
    35 0.47890941 fixed fixed 1 0 0 0.61673066 1 0.61673066 1;
  }
}

[chaneyx]

I just checked out the demo. It is disturbingly cool. 

His setup relies on having a model in Maya that has the same blendShape names as his Bebylon character. THIS IS ESSENTIAL. If you get this going, you will NOT need an in depth understanding of the .anim format. 

You will notice in your .json file strings such as eyeBlink_R, eyeWide_R, eyeLookDown_R, noseSneer_R. etc.  In order to copy that animation to Maya, you will need a face with those blendShapes. If you have that, post your progress and we can help with the next step. 

See the attached image, he has all the blendShapes set up in Maya

[harshad.asawale]

Thanks for the detailed response.

anim translate.translateY 0 2 1;

Yes, the 0 2 1 is some sort of hierarchy node numbering system, which I wanted to know. But I think this will change every time while applying for a different rig. 

0 0 auto auto 1 0 0;
1 0 auto fixed 1 0 0 0 1;

Also sometimes it's 1 0 0 at the end and sometimes it has an additional 0 1 , which is confusing.

Now it all seems that recreating the .anim file is simply not a scalable solution for applying the mocap data on any given rig. And it looks more feasible to do directly in Maya itself with a python script where the nodes can be selected to apply specific animations.

I guess i'll take the python approach. 

[harshad.asawale]

I have the fbx , with proper blendshapes setup.  PFA

[chaneyx]

The whole point of this notation is to allow things to be matched by position in the hierarchy. If you take a little time and experiment with different animExport/Import  or glance at https://download.autodesk.com/us/maya/2009help/API/class_m_anim_curve_clipboard.html you will be able to figure it out. 

Think of it is going to a theater or stadium. The first number may indicate the floor, the second the aisle, and the third number the actual seat. 

But this is not relevant for your task! Your json file gives you the attribute names which are a perfect match to your Sloth character's blendShape attribute names. So you will be matching values by name, not hierarchical position. 

[chaneyx]

I am assuming you know how to do python programming so I am only giving you a partial solution here. 

Your json file is a list of dictionaries. You can make a file reader or cut and paste it to the script editor. Each dictionary has all the keys for each frame. Those keyframes are key value pairs of attribute names to attribute values. For example "eyeBlink_R":0.630653619766235 indicates that blendShape attribute eyeBlink_R gets a value of 0.630653619766235 for that particular frame. 

Below is sample code for 1 frame. You will have to write the loop to do this for each dictionary within your file. 

import maya.cmds as cmds

frameData = {"eyeBlink_R":0.630653619766235,
"eyeWide_R":0,
"mouthLowerDown_L":0.0394449755549431,
"eyeLookDown_R":0.607172310352325,
"cheekSquint_L":0.0926333069801331,
"mouthDimple_R":0.0589954480528831,
"browInnerUp":0.215703368186951,
"eyeLookIn_L":0.000609547714702785,
"mouthPress_L":0.173792526125908,
"mouthStretch_R":0.130874514579773,
"browDown_L":0.0118590397760272,
"mouthFunnel":0.194534316658974,
"noseSneer_L":0.143848955631256,
"eyeLookOut_L":0.0232447534799576,
"eyeLookIn_R":0.133933961391449,
"mouthLowerDown_R":0.0413378998637199,
"browOuterUp_R":0.0843672901391983,
"mouthLeft":0.00630532624199986,
"cheekSquint_R":0.109647072851658,
"jawOpen":0.0494513735175133,
"eyeBlink_L":0.631030023097992,
"jawForward":0.00845738872885704,
"mouthPress_R":0.177991852164268,
"noseSneer_R":0.139457553625107,
"jawRight":0.00111576204653829,
"mouthShrugLower":0.110312908887863,
"eyeSquint_L":0.0642843469977379,
"eyeLookOut_R":7.60569946578471E-07,
"mouthFrown_L":0.146945968270302,
"cheekPuff":0.120135366916656,
"mouthStretch_L":0.160444423556328,
"mouthRollLower":0.0447048954665661,
"mouthUpperUp_R":0.085766077041626,
"mouthShrugUpper":0.129708498716354,
"eyeSquint_R":0.0651631653308868,
"mouthSmile_L":0.00668685464188457,
"eyeLookDown_L":0.609023749828339,
"eyeWide_L":0,
"mouthClose":0.0124557800590992,
"jawLeft":0.012608109973371,
"mouthDimple_L":0.0744616985321045,
"mouthFrown_R":0.145918473601341,
"mouthPucker":0.235637620091438,
"mouthRight":0.00372009864076972,
"browDown_R":0.0118923466652632,
"eyeLookUp_L":0,
"mouthSmile_R":0.00394255900755525,
"mouthUpperUp_L":0.0847973972558975,
"browOuterUp_L":0.0903196856379509,
"mouthRollUpper":0.0304965153336525,
"eyeLookUp_R":0,
"translation":[0.0122108394280076,-0.242821648716927,0.37145984172821],
"rotation":[-0.431879132986069,-0.017200656235218,-0.00822594482451677,-0.901729941368103]}

transformNode = "Sloth_Head2"
blendShapeNode = "blendShape2"

for attr in frameData.keys() :

if ( attr == "translation" ) :
# Set xform data on head
cmds.setKeyframe(transformNode, v=frameData[attr][0], at="translateX ")
cmds.setKeyframe(transformNode, v=frameData[attr][1], at="translateY ")
cmds.setKeyframe(transformNode, v=frameData[attr][2], at="translateZ ")

elif( attr == "rotation" ) :
# NOTE - you may need to convert the rotation values from radians to degrees
cmds.setKeyframe(transformNode, v=frameData[attr][0], at="rotateX")
cmds.setKeyframe(transformNode, v=frameData[attr][1], at="rotateY")
cmds.setKeyframe(transformNode, v=frameData[attr][2], at="rotateZ")

else :

# Set keyframe data on blendShapeNode
print attr
print frameData[attr]
cmds.setKeyframe( blendShapeNode , v=frameData[attr], at=attr)

[chaneyx]

How did you get the json file exported from your phone? 

[harshad.asawale]

Thanks for the python code. I know bits of python but never done it in maya.

It will be my first time. But with the code which you have given it looks pretty straightforward.

For exporting the json, i took the unity ar kit example. And modified the blendshapeprinter.cs script to get all the frame data into a json file.