Anyma Performance Capture System

The Anyma Performance Capture system, developed by DisneyResearch|Studios in Zurich, is a flexible solution for unconstrained, markerless facial performance capture. Anyma allows the actor freedom to move around, and still produces dense, consistent facial reconstruction at the quality of Medusa. The underlying actor model is built from a small number of Medusa reconstructions.
Anyma can be operated in several scenarios, ranging from a studio setup with an ADR-style booth, to on-set reconstruction with helmet-mounted cameras (HMCs). The flexibility of Anyma allows to operate from as little as one single camera, to many cameras placed around the scene.
Anyma delivers high quality, dense, facial geometry accurately tracked over time, including per-frame eye gazes. Estimated per-frame skull positions are also provided automatically, removing the need to determine the head pose or HMC stabilization manually, and per-frame jaw positions are estimated from the deforming geometry.
The video on the right shows a production test of Anyma in action. Here we see the ADR-booth scenario, where 3 film cameras observe the actor from a distance. The video is illustrated in slow motion.
The top row shows the video plates, with the reconstructed face overlaid in blue. The bottom right shows a virtual texture mapped to the actor’s face, demonstrating the accuracy of the tracking. Since the skull is also tracked automatically, we can remove the rigid head motion and show only the expression deformation in the stabilized view in bottom left.
The video on the left demonstrates automatic eye gaze tracking in Anyma. The top half shows the accuracy and stability of the gaze recovery over time, by overlaying the outer boundary of the captured iris (known as the limbus) on the input video, in a close-up view of the eyes.
The bottom half shows the same video, but this time with the complete 3D eye geometry tracked to match the actor’s eye gaze movements.
People
Anyma is the result of the hard work of many people:
Thabo Beeler
Derek Bradley
Chenglei Wu
Gaspard Zoss
Prashanth Chandran
Marco Fratarcangeli
Aurel Gruber
Pascal Bérard
Yury Gitman
Roman Cattaneo
Irene Baeza
Loïc Ciccone
Markus Gross
Research Lead
Research Lead
Research (Anatomical Face Modeling)
Research (Jaw Tracking)
Research (Landmarks)
Research (GPU Optimization)
Research (GPU Optimization)
Research (Eyes)
Software Engineering
Software Engineering
Software Engineering
Software Engineering
Lab Director
Technology
The Anyma Performance Capture System is based on the following main research projects, and is continually improved through ongoing research and technical innovation.
Anatomically-Constrained Local Deformation Model for Monocular Face Capture
July 11, 2016
ACM SIGGRAPH 2016
Chenglei Wu (ETH Zurich) Derek Bradley (Disney Research) Markus Gross (Disney Research/ETH Zurich) Thabo Beeler (Disney Research)
Accurate Markerless Jaw Tracking for Facial Performance Capture
July 12, 2019
ACM Siggraph 2019
Gaspard Zoss (Disney Research/ETH Joint PhD) Thabo Beeler (Disney Research) Markus Gross (Disney Research/ETH Zurich) Derek Bradley (Disney Research)
Attention-Driven Cropping for Very High Resolution Facial Landmark Detection
June 16, 2020
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020
Prashanth Chandran (DisneyResearch|Studios/ETH Joint PhD) Derek Bradley (DisneyResearch|Studios) Thabo Beeler (DisneyResearch|Studios) Markus Gross (DisneyResearch|Studios/ETH Zurich)
Fast Nonlinear Least Squares Optimization of Large-Scale Semi-Sparse Problems
May 25, 2020
Eurographics 2020
Marco Fratarcangeli (Chalmers University of Technology/DisneyResearch|Studios) Derek Bradley (DisneyResearch|Studios) Aurel Gruber (DisneyResearch|Studios/ETH Joint M.Sc.) Gaspard Zoss (DisneyResearch|Studios/ETH Joint PhD) Thabo Beeler (DisneyResearch|Studios)
For more information on the Anyma technology, please contact anyma@disneyresearch.com
Business Inquiries
For business inquiries, please email contact-facialcapture@ilm.com, or visit https://www.ilm.com/facial-capture/