Publications

Large Language Models (LLMs) have demonstrated impressive potential in
translating natural language (NL) instructions into program code. However, user
instructions often contain inherent ambiguities, making it challenging for LLMs
to generate code that accurately reflects the user's true intent. To address
this challenge, researchers have proposed to produce multiple candidates of the
program code and then rerank them to identify the best solution. In this paper,
we propose CodeRSA, a novel code candidate reranking mechanism built upon the
Rational Speech Act (RSA) framework, designed to guide LLMs toward more
comprehensive pragmatic reasoning about user intent. We evaluate CodeRSA using
one of the latest LLMs on a popular code generation dataset. Our experiment
results show that CodeRSA consistently outperforms common baselines, surpasses
the state-of-the-art approach in most cases, and demonstrates robust overall
performance. These findings underscore the effectiveness of integrating
pragmatic reasoning into code candidate reranking, offering a promising
direction for enhancing code generation quality in LLMs.

Transforming recorded videos into concise and accurate textual summaries is a
growing challenge in multimodal learning. This paper introduces VISTA, a
dataset specifically designed for video-to-text summarization in scientific
domains. VISTA contains 18,599 recorded AI conference presentations paired with
their corresponding paper abstracts. We benchmark the performance of
state-of-the-art large models and apply a plan-based framework to better
capture the structured nature of abstracts. Both human and automated
evaluations confirm that explicit planning enhances summary quality and factual
consistency. However, a considerable gap remains between models and human
performance, highlighting the challenges of scientific video summarization.

Research in linguistics shows that non-verbal cues, such as gestures, play a
crucial role in spoken discourse. For example, speakers perform hand gestures
to indicate topic shifts, helping listeners identify transitions in discourse.
In this work, we investigate whether the joint modeling of gestures using human
motion sequences and language can improve spoken discourse modeling in language
models. To integrate gestures into language models, we first encode 3D human
motion sequences into discrete gesture tokens using a VQ-VAE. These gesture
token embeddings are then aligned with text embeddings through feature
alignment, mapping them into the text embedding space. To evaluate the
gesture-aligned language model on spoken discourse, we construct text infilling
tasks targeting three key discourse cues grounded in linguistic research:
discourse connectives, stance markers, and quantifiers. Results show that
incorporating gestures enhances marker prediction accuracy across the three
tasks, highlighting the complementary information that gestures can offer in
modeling spoken discourse. We view this work as an initial step toward
leveraging non-verbal cues to advance spoken language modeling in language
models.

Post-hoc explanation methods for black-box models often struggle with
faithfulness and human interpretability due to the lack of explainability in
current neural models. Meanwhile, B-cos networks have been introduced to
improve model explainability through architectural and computational
adaptations, but their application has so far been limited to computer vision
models and their associated training pipelines. In this work, we introduce
B-cos LMs, i.e., B-cos networks empowered for NLP tasks. Our approach directly
transforms pre-trained language models into B-cos LMs by combining B-cos
conversion and task fine-tuning, improving efficiency compared to previous
B-cos methods. Our automatic and human evaluation results demonstrate that
B-cos LMs produce more faithful and human interpretable explanations than post
hoc methods, while maintaining task performance comparable to conventional
fine-tuning. Our in-depth analysis explores how B-cos LMs differ from
conventionally fine-tuned models in their learning processes and explanation
patterns. Finally, we provide practical guidelines for effectively building
B-cos LMs based on our findings. Our code is available at
anonymous.4open.science/r/bcos_lm.

Large Language Models (LLMs) can generate text by transferring style
attributes like formality resulting in formal or informal text. However,
instructing LLMs to generate text that when spoken, is more intelligible in an
acoustically difficult environment, is an under-explored topic. We conduct the
first study to evaluate LLMs on a novel task of generating acoustically
intelligible paraphrases for better human speech perception in noise. Our
experiments in English demonstrated that with standard prompting, LLMs struggle
to control the non-textual attribute, i.e., acoustic intelligibility, while
efficiently capturing the desired textual attributes like semantic equivalence.
To remedy this issue, we propose a simple prompting approach,
prompt-and-select, which generates paraphrases by decoupling the desired
textual and non-textual attributes in the text generation pipeline. Our
approach resulted in a 40% relative improvement in human speech perception, by
paraphrasing utterances that are highly distorted in a listening condition with
babble noise at a signal-to-noise ratio (SNR) -5 dB. This study reveals the
limitation of LLMs in capturing non-textual attributes, and our proposed method
showcases the potential of using LLMs for better human speech perception in
noise.

Diffusion models have recently achieved remarkable advancements in terms of
image quality and fidelity to textual prompts. Concurrently, the safety of such
generative models has become an area of growing concern. This work introduces a
novel type of jailbreak, which triggers T2I models to generate the image with
visual text, where the image and the text, although considered to be safe in
isolation, combine to form unsafe content. To systematically explore this
phenomenon, we propose a dataset to evaluate the current diffusion-based
text-to-image (T2I) models under such jailbreak. We benchmark nine
representative T2I models, including two close-source commercial models.
Experimental results reveal a concerning tendency to produce unsafe content:
all tested models suffer from such type of jailbreak, with rates of unsafe
generation ranging from 8\% to 74\%. In real-world scenarios, various filters
such as keyword blocklists, customized prompt filters, and NSFW image filters,
are commonly employed to mitigate these risks. We evaluate the effectiveness of
such filters against our jailbreak and found that, while current classifiers
may be effective for single modality detection, they fail to work against our
jailbreak. Our work provides a foundation for further development towards more
secure and reliable T2I models.

Non-verbal communication often comprises of semantically rich gestures that
help convey the meaning of an utterance. Producing such semantic co-speech
gestures has been a major challenge for the existing neural systems that can
generate rhythmic beat gestures, but struggle to produce semantically
meaningful gestures. Therefore, we present RAG-Gesture, a diffusion-based
gesture generation approach that leverages Retrieval Augmented Generation (RAG)
to produce natural-looking and semantically rich gestures. Our neuro-explicit
gesture generation approach is designed to produce semantic gestures grounded
in interpretable linguistic knowledge. We achieve this by using explicit domain
knowledge to retrieve exemplar motions from a database of co-speech gestures.
Once retrieved, we then inject these semantic exemplar gestures into our
diffusion-based gesture generation pipeline using DDIM inversion and retrieval
guidance at the inference time without any need of training. Further, we
propose a control paradigm for guidance, that allows the users to modulate the
amount of influence each retrieval insertion has over the generated sequence.
Our comparative evaluations demonstrate the validity of our approach against
recent gesture generation approaches. The reader is urged to explore the
results on our project page.