Non-Rigid Model Acquisition
Rigid and chain-based models can cover coarse full-body motion, but ignore subtle non-rigid deformations. Problem areas are facial and lip-deformations, complex torso motion, muscle-deformations on arms and legs, and shoe deformations. With this new project, we started developing techniques that can pick-up those subtleties in 3D. We can recover non-rigid basis shape models from monocular views in 3D (Non-rigid Structure from Motion).

                                                                          

Learning Non-Rigid 3D Shape from 2D Motion:

How can we reconstruct non-rigid 3D shape from 2D motion with as little prior knowledge as possible?

We show that, by assuming that the 3D shape is drawn from a parametric distribution, we can simultaneously learn the non-rigid 3D shape, the rigid motion  and the parameters of the shape distribution.

 

Space-Time Tracking:

Our new model-free tracking technique exploits a space-time rank constraint to capture non-rigid motion. Here we demonstrate the algorithm by tracking Charlie Chaplin’s facial expressions from an old film footage.

 

 

 

 
Integrated low-level tracking and non-rigid 3D reconstruction:
Here are two mpeg movies that show our new low-level tracking and reconstruction on a deforming shoe and torso. The method recovers automatically the feature point-tracks.

 
Acquiring non-rigid structure and motion from point tracks:

Here is a mpeg movie of a single view recording of John F. Kennedy face and it's 3D reconstructed non-rigid shapes. To the best of our knowledge, this is the first model-free system that can do non-rigid structure-from-motion from monocular views.

Publications

Learning Non-Rigid 3D Shape from 2D Motion
Lorenzo Torresani, Aaron Hertzmann and Christoph Bregler
Proc. NIPS 2003 (pdf)

Space-Time Tracking
Lorenzo Torresani and Christoph Bregler
Proc. ECCV 2002 (pdf, ps.gz)

CVPR 2001 Best Student Paper Award:
Tracking and Modeling Non-Rigid Objects with Rank Constraints
Lorenzo Torresani, Danny Yang, Gene Alexander, and Christoph Bregler
Proc. IEEE CVPR 2001 (pdf, ps.gz)

Recovering Non-Rigid 3D Shape from Image Streams
Christoph Bregler, Aaron Hertzmann, and Henning Biermann
Proc. IEEE CVPR 2000 (pdf, ps.gz)