Anna Khoreva (Post-Doc)

The estimation of 3D human poses from images has progressed tremendously over
the last few years as measured on standard benchmarks. However, performance in
the open world remains underexplored, as current benchmarks cannot capture its
full extent. Especially in safety-critical systems, it is crucial that 3D pose
estimators are audited before deployment, and their sensitivity towards single
factors or attributes occurring in the operational domain is thoroughly
examined. Nevertheless, we currently lack a benchmark that would enable such
fine-grained analysis. We thus present STAGE, a GenAI data toolkit for auditing
3D human pose estimators. We enable a text-to-image model to control the 3D
human body pose in the generated image. This allows us to create customized
annotated data covering a wide range of open-world attributes. We leverage
STAGE and generate a series of benchmarks to audit the sensitivity of popular
pose estimators towards attributes such as gender, ethnicity, age, clothing,
location, and weather. Our results show that the presence of such naturally
occurring attributes can cause severe degradation in the performance of pose
estimators and leads us to question if they are ready for open-world
deployment.

Despite tremendous progress in the field of text-to-video (T2V) synthesis,
open-sourced T2V diffusion models struggle to generate longer videos with
dynamically varying and evolving content. They tend to synthesize quasi-static
videos, ignoring the necessary visual change-over-time implied in the text
prompt. At the same time, scaling these models to enable longer, more dynamic
video synthesis often remains computationally intractable. To address this
challenge, we introduce the concept of Generative Temporal Nursing (GTN), where
we aim to alter the generative process on the fly during inference to improve
control over the temporal dynamics and enable generation of longer videos. We
propose a method for GTN, dubbed VSTAR, which consists of two key ingredients:
1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis
based on the original single prompt leveraging LLMs, which gives accurate
textual guidance to different visual states of longer videos, and 2) Temporal
Attention Regularization (TAR) - a regularization technique to refine the
temporal attention units of the pre-trained T2V diffusion models, which enables
control over the video dynamics. We experimentally showcase the superiority of
the proposed approach in generating longer, visually appealing videos over
existing open-sourced T2V models. We additionally analyze the temporal
attention maps realized with and without VSTAR, demonstrating the importance of
applying our method to mitigate neglect of the desired visual change over time.

Emerging large-scale text-to-image generative models, e.g., Stable Diffusion
(SD), have exhibited overwhelming results with high fidelity. Despite the
magnificent progress, current state-of-the-art models still struggle to
generate images fully adhering to the input prompt. Prior work, Attend &
Excite, has introduced the concept of Generative Semantic Nursing (GSN), aiming
to optimize cross-attention during inference time to better incorporate the
semantics. It demonstrates promising results in generating simple prompts,
e.g., "a cat and a dog". However, its efficacy declines when dealing with more
complex prompts, and it does not explicitly address the problem of improper
attribute binding. To address the challenges posed by complex prompts or
scenarios involving multiple entities and to achieve improved attribute
binding, we propose Divide & Bind. We introduce two novel loss objectives for
GSN: a novel attendance loss and a binding loss. Our approach stands out in its
ability to faithfully synthesize desired objects with improved attribute
alignment from complex prompts and exhibits superior performance across
multiple evaluation benchmarks.

Much progress has been made in image and video segmentation
over the last years. To a large extent, the success can be attributed to
the strong appearance models completely learned from data, in particular
using deep learning methods. However,to perform best these methods require
large representative datasets for training with expensive pixel-level
annotations, which in case of videos are prohibitive to obtain. Therefore,
there is a need to relax this constraint and to consider alternative forms
of supervision, which are easier and cheaper to collect. In this thesis,
we aim to develop algorithms for learning to segment in images and videos
with different levels of supervision.
First, we develop approaches for training convolutional networks with weaker
forms of supervision, such as bounding boxes or image labels, for object
boundary estimation and semantic/instance labelling tasks. We propose to
generate pixel-level approximate groundtruth from these weaker forms of
annotations to train a network, which allows to achieve high-quality
results comparable to the full supervision quality without any
modifications of the network architecture or the training procedure.
Second, we address the problem of the excessive computational and memory
costs inherent to solving video segmentation via graphs. We propose
approaches to improve the runtime and memory efficiency as well as the
output segmentation quality by learning from the available training data
the best representation of the graph. In particular, we contribute with
learning must-link constraints, the topology and edge weights of the graph
as well as enhancing the graph nodes - superpixels - themselves.
Third, we tackle the task of pixel-level object tracking and address the
problem of the limited amount of densely annotated video data for training
convolutional networks. We introduce an architecture which allows training
with static images only and propose an elaborate data synthesis scheme
which creates a large number of training examples close to the target
domain from the given first frame mask. With the proposed techniques we
show that densely annotated consequent video data is not necessary to
achieve high-quality temporally coherent video segmentationresults.
In summary, this thesis advances the state of the art in weakly supervised
image segmentation, graph-based video segmentation and pixel-level object
tracking and contributes with the new ways of training convolutional
networks with a limited amount of pixel-level annotated training data.

Convolutional networks reach top quality in pixel-level object tracking but

require a large amount of training data (1k ~ 10k) to deliver such results. We

propose a new training strategy which achieves state-of-the-art results across

three evaluation datasets while using 20x ~ 100x less annotated data than

competing methods. Instead of using large training sets hoping to generalize

across domains, we generate in-domain training data using the provided

annotation on the first frame of each video to synthesize ("lucid dream")

plausible future video frames. In-domain per-video training data allows us to

train high quality appearance- and motion-based models, as well as tune the

post-processing stage. This approach allows to reach competitive results even

when training from only a single annotated frame, without ImageNet

pre-training. Our results indicate that using a larger training set is not

automatically better, and that for the tracking task a smaller training set

that is closer to the target domain is more effective. This changes the mindset

regarding how many training samples and general "objectness" knowledge are

required for the object tracking task.

State-of-the-art learning based boundary detection methods require extensive

training data. Since labelling object boundaries is one of the most expensive

types of annotations, there is a need to relax the requirement to carefully

annotate images to make both the training more affordable and to extend the

amount of training data. In this paper we propose a technique to generate

weakly supervised annotations and show that bounding box annotations alone

suffice to reach high-quality object boundaries without using any

object-specific boundary annotations. With the proposed weak supervision

techniques we achieve the top performance on the object boundary detection

task, outperforming by a large margin the current fully supervised

state-of-the-art methods.

Graph-based video segmentation methods rely on superpixels as starting point.

While most previous work has focused on the construction of the graph edges and

weights as well as solving the graph partitioning problem, this paper focuses

on better superpixels for video segmentation. We demonstrate by a comparative

analysis that superpixels extracted from boundaries perform best, and show that

boundary estimation can be significantly improved via image and time domain

cues. With superpixels generated from our better boundaries we observe

consistent improvement for two video segmentation methods in two different

datasets.