Advanced Video Processing

Christian Theobalt

Advanced Video Processing

Image processing methods are important tools during the post-processing of photographs. Many standard tools for image processing provide algorithms to fulfil typical post-processing tasks, such as standard filters for noise removal or contrast enhancement.

Post-processing of videos is a much more challenging task. Videos are not merely a temporal sequence of individual images. Modifications to videos during post-processing, therefore, need not only be spatially consistent, that is, consistent within a single image, they also need to be consistent over time, that is, over many subsequent frames. Many commercially available video editing tools, however, are based on exactly this assumption that videos can be treated like sequences of individual images. Post-Processing operations with such tools are consequently limited to the application of simple filters
over a temporal window.

Many post-processing tasks in professional video and movie productions are much more complex: Entire scene elements, such as particular actors or supporting crew, may need to be removed or manipulated in a shot, or their positions in a shot may need to be changed. No existing video fi ltering approach would even come close to solving such a task automatically. The consequence is that such editing tasks are usually undertaken through time-consuming manual editing of individual pixels, which can easily take several weeks, even for short videos.

Further, today’s personal and online video collections are extremely large. Browsing such collections is a challenge, as current approaches are often based on unreliable user-based text annotation. Ideally, one would want more semantically-based ways of exploration that exploit space-time and content relations between videos, such as the fact that videos were taken in the same environment, show a similar scene, were taken at a similar time, or deal with a similar topic. The precondition for this would be the ability to automatically extract such relations, which is a highly challenging and unresolved problem. We investigate the algorithmic foundations of both of the above problem areas.