topic Re: Live stream facial motion capture with Iphone x in MotionBuilder Forum

Live stream facial motion capture with Iphone x

Anonymous — Wed, 22 Nov 2017 05:18:52 GMT

What would I need to live stream facial mocap data with the iphone x depth camera? There is already a blendshape dictionary created within the phone according to ARkit docs but I don't see how one would stream it into motionbuilder to drive a 3d face. I'd like to use it simultaneously with perception neuron streaming. I'd love any suggestions or can anyone point me in the right direction?

Re: Live stream facial motion capture with Iphone x

KikoBarahona — Thu, 23 Nov 2017 02:10:54 GMT

I'm guessing you would have to map those blendshapes to an Actor/Character face in motionbuilder.

We used Character face to drive a cluster based face rig in motionbuilder with Faceshift and a Kinect camera.

Re: Live stream facial motion capture with Iphone x

Anonymous — Thu, 30 Nov 2017 00:31:22 GMT

You're awesome for responding KikoBarahona. Faceshift and Kinect worked great for us too a couple years back, but they had a motionbuilder plugin. I just hoped that someone might have made a plugin that I could use with the iphone x or another depth mobile phone. I don't know how to stream the blendshapes without a plugin but I can't imagine it would be too hard. Also curious if the mobu faceshift plugin would work with a minor tweak or something if anybody knows.

To put a phone on a Helmet to stream facial mocap while simultaneously streaming body with something like perception neuron would make me a very happy person!

Re: Live stream facial motion capture with Iphone x

KikoBarahona — Mon, 04 Dec 2017 21:17:52 GMT

There's an article on Cartoon Brew about doing facial animation with the iPhone.

Using Apple ARKit and Houdini (SideFX ships a free version of Houdini, btw)

Hope that helps.

Re: Live stream facial motion capture with Iphone x

taraftarium820 — Mon, 12 Aug 2024 04:00:13 GMT

To live stream facial motion capture data from the iPhone X's depth camera into MotionBuilder, you’ll need to set up a system to capture and transfer the data from the ARKit framework. Here’s a step-by-step approach to help you get started:

ARKit Integration: Begin by utilizing ARKit to capture facial motion data. You can create a custom iOS app to access ARKit’s facial tracking features and obtain the blendshape coefficients.
Data Streaming: To stream the data, you’ll need to implement a mechanism in your iOS app that sends the blendshape data over a network connection. You might use a socket connection (e.g., WebSockets) or a custom protocol to transmit the data to your computer.
Receiver Setup: On your computer, set up a receiver that listens for incoming data from the iOS app. This could be a custom application or script that parses the received blendshape data.
MotionBuilder Integration: To integrate with MotionBuilder, you’ll need to use its SDK or scripting capabilities (Python or MEL). Write a script to interpret the incoming blendshape data and map it to the corresponding 3D facial controls in MotionBuilder.
Simultaneous Streaming: For simultaneous streaming with Perception Neuron, ensure that your system can handle multiple data streams. You might need to create a unified interface or middleware that combines data from both sources before sending it to MotionBuilder.
Testing and Calibration: Test your setup to ensure accurate data transfer and synchronization. Adjust the mapping as needed to ensure that the facial motion is correctly reflected in MotionBuilder.

For further guidance, you might want to check out developer forums, ARKit documentation, or MotionBuilder’s API references. Engaging with communities or seeking advice from experts who have worked on similar integrations can also provide valuable insights. Hope it helps:

Re: Live stream facial motion capture with Iphone x

_neill_ — Mon, 16 Sep 2024 21:16:00 GMT

There is a Face Cap app for iPhones, which you can use to record or stream data in real-time https://apps.apple.com/us/app/face-cap-motion-capture/id1373155478

And there is a open source device plugin for the facecap stream as part of OpenMoBu, you can find it in releases section of github repository - https://github.com/Neill3d/OpenMoBu/releases/tag/2024