Excited to announce that I'll be starting my PhD at @LTIatCMU this Fall working on generation and controlling LMs 🥳! Big thank you to my mentors + letter writers @mmitchell_ai, @_DougDowney and @mattthemathman and all my collaborators at @allen_ai for their invaluable support ❤️.

RT @bearseascape: 🚨New #interpretability paper with @nsubramani23 :🕵️Model Internal Sleuthing: Finding Lexical Identity and Inflectional Mo….

🚨 Check out our new #interpretability paper: 🕵🏽 Model Internal Sleuthing led by the amazing @bearseascape who is an undergrad at @SCSatCMU @LTIatCMU!.

Excited to announce that I started at @googlecloud as a student researcher last month working with @hamidpalangi on actionable #interpretability 🔍 to build better tool using #agents ⚒️🤖.

Presenting this today at the poster session at #NAACL2025!. Come chat about interpretability, trustworthiness, and tool-using agents!. 🗓️ - Thursday May 1st (today).📍 - Hall 3.🕑 - 200-330pm.

At #NAACL2025 🌵till Sunday! Love to chat about interpretability, understanding model internals, and finding vegan food 🥬.

Come to our poster in Albuquerque on Thursday 2-330pm in the interpretability & analysis section! . Paper: Code (coming soon): 🧵/🧵.

MICE 🐭: .🎯 - significantly beats baselines on expected tool-calling utility, especially in high risk scenarios .✅ - matches expected calibration error of baselines .✅ - is sample efficient .✅ - generalizes zeroshot to unseen tools . 5/🧵

Calibration is not sufficient: both an oracle and a model that just predicts the base rate are perfectly calibrated🤦🏽‍♂️. We develop a new metric expected tool-calling utility 🛠️to measure the utility of deciding whether or not to execute a tool call via a confidence score! . 4/🧵.

We propose 🐭 MICE to better assess confidence when calling tools: .1️⃣ decode from each intermediate layer of an LM .2️⃣ compute similarity scores between each layer’s generation and the final output. 3️⃣ train a probabilistic classifier on these features . 3/🧵

1️⃣ Tool-using agents need to be useful and safe as they take actions in the world .2️⃣ Language models are poorly calibrated .🤔 Can we use model internals to better calibrate language models to make tool-using agents safer and more useful? . 2/🧵.

🚀 Excited to share a new interp+agents paper: 🐭🐱 MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools appearing at #NAACL2025. This was work done @Microsoft last summer with @adveisner @justinsvegliato @ben_vandurme @ysu_nlp @sammthomson . 1/🧵

RT @claranahhh: Building/customizing your own LLM? You'll want to curate training data for it, but how do you know what makes the data good….

RT @apoorvkh: Wondering how long it takes to train a 1B-param LM from scratch on your GPUs? 🧵. See our paper to learn about the current sta….

RT @TEKnologyy: 🚀 Excited to announce our latest paper, "Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation"….

RT @JesseDodge: Congrats to our team for winning two paper awards at #ACL2024!. OLMo won the Best Theme Paper award, and Dolma won a Best R….

Presenting our paper "Evaluating Personal Information Parroting in Language Models" joint work with @GhateKshitish @MonaDiab77 from @LTIatCMU at the @trustnlp poster session now-330p today. Poster 64! #NAACL2024

Great talk by @abeirami at @trustnlp on algorithmic LM alignment with a recipe to approach alignment problems #NAACL2024

Great talk on linear relational concepts in LLMs on work by @chanindav, Anthony Hunter, and @oanacamb from @ucl_nlp #NAACL2024. Paper:

Cool interpretability poster from @jack_merullo_ @Brown_NLP on showing that LMs do word2vsc style arithmetic in their latent spaces for analogy type tasks #NAACL2024

In Mexico City for #NAACL2024 through next Sunday! Hoping to chat with folks thinking about model interpretability research 🔍, explore vegan food 🌮, learn about the indigenous cultures 🐍🐆🦅 of the region, and explore the history of modern Mexico 🇲🇽 - DM me!.