Automatic Non-Rigid 3D Modeling from Video

We present a robust framework for estimating non-rigid 3D shape and motion in video sequences. Given an input video sequence, and a user-specified region to reconstruct, the algorithm automatically solves for the 3D time-varying shape and motion of the object, and estimates which pixels are outliers, while learning all system parameters, including a PDF over non-rigid deformations. There are no user-tuned parameters (other than initialization); all parameters are learned by maximizing the likelihood of the entire image stream. We apply our method to both rigid and non-rigid shape reconstruction, and demonstrate it in challenging cases of occlusion and variable illumination.

paper: Automatic Non-Rigid 3D Modeling from Video, Lorenzo Torresani and Aaron Hertzmann, to appear, ECCV 2004 (pdf) errata

See also our work on non-rigid structure-from-motion presented at NIPS 2003.

results:

Tracking and 3D reconstruction of non-rigid human motion. Features were selected automatically in the first frame. Points are colored according to their estimated outlier probability: green for completely valid pixels, red for outliers and occluded points

Our robust tracking method applied to a sequence of mostly-rigid face/head motion. Points from the left side of the subject's face are occluded for more than 50% of the sequence.

Tracking and 3D reconstruction from a bullfight sequence, taken from the movie Talk To Her.

Additional results