Markerless Performance Capture

Nils Hasler & Carsten Stoll

Markerless Performance Capture

The ability to capture detailed mo- tion data from real people or from technical processes is of critical importance in many application areas of commerce and science. In medicine or biomechanics, motion capture is used to measure the success of a treatment or the efficiency of a sequence of movements. Motion capture is also applied in product design and in ergonomics research, for example to draw conclusions about the usability of technical devices. Today it is even used in onboard systems in vehicles, for example to recognize pedestrians and their movements in the vicinity of an automobile. Moreover, motion capture processes are used in the film and computer game industries. Virtual actors (avatars) are fundamental components of special effects and virtual 3D worlds.

With respect to commercially-available motion capture methods, marker- based optical methods are, as a general rule, the state of the art in industry. These processes are invasive, since certain markers need to be brought into the scene. The use of these systems is costly and time-consuming. In many cases these methods cannot be applied, for example when it is not possible to actively intervene in the scene.

With support from the Federal Ministry of Economics and Technology (BMWi) through its EXIST Transfer of Research (EXIST-Forschungstransfer) program, our team, consisting of Dr. Nils Hasler, Dr. Carsten Stoll, MsC Michal Richter, and our mentor, Prof. Dr. Christian Theobalt, develops software for the markerless reconstruction of human movements from normal video images. First, the skeletal movement is computed using a new, very robust and also fast approach. This represents the video images and the tracked person through implicit functions, and then attempts to reconcile these. In contrast to industrial methods, this approach requires neither markers nor explicit silhouettes. Thus, it is possible for us to capture the movements of people wearing everyday clothing. This is especially important for motion capture of athletes or virtual actors.

The markerless tracking method we have developed is qualitatively sophisticated and comparable to marker-based approaches. It is, however, clearly more flexible and works without as many restrictive constraints. Thus, the new method functions without a green screen, and without synchronization of the cameras, while at the same time being faster than all of the markerless tracking processes from the literature. In addition, the characteristics that (with a greatly increased runtime) gave earlier approaches an accuracy advantage, for example feature tracking and implicit stereo reconstruction, can be integrated in the new method. In this way, in addition to the skeletal movement, a 3D surface reconstruction of the person can be captured (so-called performance capture). Here a complex optimization consisting of a combination of multi-view stereo and shape-from-shading is carried out. This enables the temporal reconstruction of the movements of loose clothing or folds.