Mina Lee (Stanford) - Designing and Evaluating Language Models for Human Interaction

In this talk, I will present my research on designing and evaluating language models (LMs) for human-LM interaction. Despite the recent advancements in LMs, most LMs are not optimized for, nor are they evaluated on, real-world usage with human interaction. To address this, I will first describe how we can support human editing needs by enabling any LM to perform text infilling at any position in a document (i.e., “fill in the blanks”). I will then introduce CoAuthor, a platform for capturing human-LM interaction in collaborative writing as rich, replayable, keystroke-level interaction traces. With the platform, I demonstrate how collecting a large interaction dataset and analyzing the traces provide unique insights into LM capabilities regarding language, ideation, and collaboration. Lastly, I will propose a new framework, HALIE (Human-AI Language-based Interaction Evaluation), that defines the components of interactive systems and evaluation metrics for human-LM interaction beyond writing. I will conclude by discussing open challenges and future directions in this field.