BinaryVR: Facial Capture in High Fidelity

The BinaryVR face tracking device attaches to your HMD and gives your High Fidelity avatar facial expressions.

BinaryVR’s new real time facial tracking system, which is now available for pre-order, has been integrated into High Fidelity, providing real time mouth movement tracking for avatars. The technology is part of a growing range of accessories design to work with and improve the Head Mounted Display (HMD) experience.

Check out how well BinaryVR adds another level of human communication to VR:

A little High Fidelity history: before there were HMDs, we enabled rich facial expressions for avatars using either the Primesense depth camera or a regular web camera with some additional processing. The video below — a lip-sync of Bohemian Rhapsody — is a great demonstration of Primesense face capture. Another example below is our 2015 funding announcement video. It shows capture for three of our staff with depth cameras trained on their faces. As you can see, both technologies add a lot to what you can convey as an avatar, and to the social VR experience.

Enter HMDs, such as the Oculus Rift and the HTC Vive. Although these devices are amazing, they block the user’s face, making it impossible to capture facial expressions with the methods described above. We lost the ability to capture facial expressions directly. As a temporary workaround, social VR solutions have instead animated avatar mouths based on audio levels.

Happily, over the last couple of years, new research has begun to suggest ways of capturing facial expressions from an HMD. In particular, Hao Li at USC worked with Oculus on this paper, showing how a mounted depth camera on an HMD could be used to capture facial movements. Some of these ideas are now coming to market.

The BinaryVR team has brought together expression capture experience from ILM with research in computer vision and machine learning to make some of this research a reality. We’re excited to see their product come to market!