Publications - Current Year

Despite recent progress, large language models (LLMs) still face the
challenge of appropriately reacting to the intricacies of social and cultural
conventions. This paper presents MANGO, a methodology for distilling
high-accuracy, high-recall assertions of cultural knowledge. We judiciously and
iteratively prompt LLMs for this purpose from two entry points, concepts and
cultures. Outputs are consolidated via clustering and generative summarization.
Running the MANGO method with GPT-3.5 as underlying LLM yields 167K
high-accuracy assertions for 30K concepts and 11K cultures, surpassing prior
resources by a large margin. For extrinsic evaluation, we explore augmenting
dialogue systems with cultural knowledge assertions. We find that adding
knowledge from MANGO improves the overall quality, specificity, and cultural
sensitivity of dialogue responses, as judged by human annotators. Data and code
are available for download.

Methods for relation extraction from text mostly focus on high precision, at
the cost of limited recall. High recall is crucial, though, to populate long
lists of object entities that stand in a specific relation with a given
subject. Cues for relevant objects can be spread across many passages in long
texts. This poses the challenge of extracting long lists from long texts. We
present the L3X method which tackles the problem in two stages: (1)
recall-oriented generation using a large language model (LLM) with judicious
techniques for retrieval augmentation, and (2) precision-oriented
scrutinization to validate or prune candidates. Our L3X method outperforms
LLM-only generations by a substantial margin.