The tool-calling decision-making dataset I developed during my internship at NVIDIA is now out! Catch me presenting it at NAACL with @ameyasm1154 or see the 🧵 for details.

You can check out our paper at We'll also have a poster at NENLP on April 11th where you can come and ask questions 😊.

What do you want to know?.

This suggests that humans and LLMs really do solve this task using composition, if we're willing to accept behavioral evidence as evidence of compositionality—see Kate McCurdy's excellent compositionality survey

We build a computational model of analogical reasoning using word embedding similarity, including Llama 3 embeddings and also run an experiment asking humans to reason by analogy. We find that not all of our data can be handled by analogy!

Many linguists assume that humans compute the meaning and inferences of multi-word phrases by composition of meaning. But working with LLMs raises the possibility that LLMs (and humans!) are actually solving this task by analogy, like so:

New paper with @najoungkim and @TeaAnd_OrCoffee! .We previously found that humans and LLMs generalize to novel adjective-noun inferences (e.g. "Is a homemade currency still a currency?") .Turns out analogy isn't enough to generalize, so it's likely evidence of composition! 🧵

We'll also be presenting this work at GenBench (co-located with EMNLP) on 11/16, so come catch us then!.

New paper with @najoungkim and @TeaAnd_OrCoffee testing if LLMs can draw adjective-noun inferences like humans! Turns out they often can, and even generalize to unseen combinations. But they're more optimistic about "artificial intelligence" than humans.

New preprint with @najoungkim & @TeaAnd_OrCoffee on fake reefs and other cases of (novel) adjective-noun composition: Whether a fake N is an N or not depends on noun + context, but people handle novel AN pairs just fine 🙂.Stay tuned for results on LLMs!