Publications

Post-hoc explanation methods for black-box models often struggle with
faithfulness and human interpretability due to the lack of explainability in
current neural models. Meanwhile, B-cos networks have been introduced to
improve model explainability through architectural and computational
adaptations, but their application has so far been limited to computer vision
models and their associated training pipelines. In this work, we introduce
B-cos LMs, i.e., B-cos networks empowered for NLP tasks. Our approach directly
transforms pre-trained language models into B-cos LMs by combining B-cos
conversion and task fine-tuning, improving efficiency compared to previous
B-cos methods. Our automatic and human evaluation results demonstrate that
B-cos LMs produce more faithful and human interpretable explanations than post
hoc methods, while maintaining task performance comparable to conventional
fine-tuning. Our in-depth analysis explores how B-cos LMs differ from
conventionally fine-tuned models in their learning processes and explanation
patterns. Finally, we provide practical guidelines for effectively building
B-cos LMs based on our findings. Our code is available at
anonymous.4open.science/r/bcos_lm.

Large Language Models (LLMs) can generate text by transferring style
attributes like formality resulting in formal or informal text. However,
instructing LLMs to generate text that when spoken, is more intelligible in an
acoustically difficult environment, is an under-explored topic. We conduct the
first study to evaluate LLMs on a novel task of generating acoustically
intelligible paraphrases for better human speech perception in noise. Our
experiments in English demonstrated that with standard prompting, LLMs struggle
to control the non-textual attribute, i.e., acoustic intelligibility, while
efficiently capturing the desired textual attributes like semantic equivalence.
To remedy this issue, we propose a simple prompting approach,
prompt-and-select, which generates paraphrases by decoupling the desired
textual and non-textual attributes in the text generation pipeline. Our
approach resulted in a 40% relative improvement in human speech perception, by
paraphrasing utterances that are highly distorted in a listening condition with
babble noise at a signal-to-noise ratio (SNR) -5 dB. This study reveals the
limitation of LLMs in capturing non-textual attributes, and our proposed method
showcases the potential of using LLMs for better human speech perception in
noise.

Diffusion models have recently achieved remarkable advancements in terms of
image quality and fidelity to textual prompts. Concurrently, the safety of such
generative models has become an area of growing concern. This work introduces a
novel type of jailbreak, which triggers T2I models to generate the image with
visual text, where the image and the text, although considered to be safe in
isolation, combine to form unsafe content. To systematically explore this
phenomenon, we propose a dataset to evaluate the current diffusion-based
text-to-image (T2I) models under such jailbreak. We benchmark nine
representative T2I models, including two close-source commercial models.
Experimental results reveal a concerning tendency to produce unsafe content:
all tested models suffer from such type of jailbreak, with rates of unsafe
generation ranging from 8\% to 74\%. In real-world scenarios, various filters
such as keyword blocklists, customized prompt filters, and NSFW image filters,
are commonly employed to mitigate these risks. We evaluate the effectiveness of
such filters against our jailbreak and found that, while current classifiers
may be effective for single modality detection, they fail to work against our
jailbreak. Our work provides a foundation for further development towards more
secure and reliable T2I models.

Non-verbal communication often comprises of semantically rich gestures that
help convey the meaning of an utterance. Producing such semantic co-speech
gestures has been a major challenge for the existing neural systems that can
generate rhythmic beat gestures, but struggle to produce semantically
meaningful gestures. Therefore, we present RAG-Gesture, a diffusion-based
gesture generation approach that leverages Retrieval Augmented Generation (RAG)
to produce natural-looking and semantically rich gestures. Our neuro-explicit
gesture generation approach is designed to produce semantic gestures grounded
in interpretable linguistic knowledge. We achieve this by using explicit domain
knowledge to retrieve exemplar motions from a database of co-speech gestures.
Once retrieved, we then inject these semantic exemplar gestures into our
diffusion-based gesture generation pipeline using DDIM inversion and retrieval
guidance at the inference time without any need of training. Further, we
propose a control paradigm for guidance, that allows the users to modulate the
amount of influence each retrieval insertion has over the generated sequence.
Our comparative evaluations demonstrate the validity of our approach against
recent gesture generation approaches. The reader is urged to explore the
results on our project page.