Instruction tuning unlocks incredible skills in LLMs, but at a cost: they become dangerously overconfident. You face a choice: a well-calibrated base model or a capable but unreliable instruct model. What if you didn't have to choose? What if you could navigate the trade-off?

@_AndrewZhao That’s where cross-tokenizer distillation comes in handy!

⚠️ Only 2 days remaining to apply for a postdoc at @EdinburghNLP! ⚠️

📢 New #COLM2025 paper 📢 Standard benchmarks give every LLM the same questions. This is like testing 5th graders and college seniors with *one* exam! 🥴 Meet Fluid Benchmarking, a capability-adaptive eval method delivering lower variance, higher validity, and reduced cost. 🧵

I am looking for a 2-year 𝗽𝗼𝘀𝘁𝗱𝗼𝗰 to work on efficient foundation models at @InfAtEd and @EPCCed! This is part of the @ARIA_research funding for Scaling Compute: AI at 1/1000th the cost

I've been awarded a Starting Grant from @ERC_Research! As part of AToM-FM ⚛️, I'll study efficient architectures for foundation models with end-to-end tokenisation and adaptive+permanent memory Building a greener, more democratic AI

Evaluating language models is tricky, how do we know if our results are real, or due to random chance? We find an answer with two simple metrics: signal, a benchmark’s ability to separate models, and noise, a benchmark’s random variability between training steps 🧵

This one is also pretty good

- Retrofitting Large Language Models with Dynamic Tokenization @licwu @bminixhofer - Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art @ChenLiu47008770 - DARE: Diverse Visual Question Answering with Robustness Evaluation Hannah Sterz, @licwu

In a weird way, I think this work supports @bminixhofer's idea of tokenizer transfer for a single model by equalizing the log prob of spans of text from a corpus (rather than trying to equalize the lob prob of some huge tree of continuations)

New paper & surprising result. LLMs transmit traits to other models via hidden signals in data. Datasets consisting only of 3-digit numbers can transmit a love for owls, or evil tendencies. 🧵

We built sparse-frontier — a clean abstraction that lets you focus on your custom sparse attention implementation while automatically inheriting vLLM’s optimizations and model support. As a PhD student, I've learned that sometimes the bottleneck in research isn't ideas — it's

There are also many other things which didn’t have space in this thread. This includes using our method to ensemble heterogeneous models and improve zero-shot tokenizer transfer. Details on this are in the paper!

🤗Models: https://t.co/hXnMWMS6pl 🧑‍💻Code: https://t.co/W9TalIKfid 📜Paper:

This allows cheaply creating new byte-level models, instead of needing to expensively train from scratch (where many things can go wrong). It could finally allow us to get rid of subword tokenization. Our paper, models and easy-to-use code are public today.

The resulting byte-level models perform competitively on benchmarks even though we are using a very short (~300M subword token) training regime.

Distilling across tokenizers was previously only possible heuristically between very similar tokenizers. We introduce a more capable and principled method which minimizes the likelihood difference between aligned chunks of tokens of the teacher and the student.

BLT showed that it is possible to create a byte-level model by retrofitting an existing subword-based model. However, they simply train via next-byte prediction. This is (as we show) very destructive. We can do much better if we instead *distil* from subwords to bytes.

We achieved the first instance of successful subword-to-byte distillation in our (just updated) paper. This enables creating byte-level models at a fraction of the cost of what was needed previously. As a proof-of-concept, we created byte-level Gemma2 and Llama3 models. 🧵

Excited to see our study on linguistic generalization in LLMs featured by @UniofOxford News!