I recently gave a tutorial on knowledge distillation for LLMs, explaining the mathematical derivations behind the commonly used methods. Sharing the slides here given the recent interest in this topic.

RT @GoogleDeepMind: An advanced version of Gemini with Deep Think has officially achieved gold medal-level performance at the International….

RT @yong_zhengxin: I wrote up this post about how we should **unify RL and next-token-prediction** based on my perspective how humans learn….

RT @RotekSong: Kimi K2 is here! The first big beautiful model purpose-built for agentic capabilities is now open-source! Agent RL, ready fo….

RT @jacobmbuckman: The age of transformers is ending. the dawn of linear-cost architectures is upon us. Power Attention replaces Flash Att….

RT @george__wing: Great talk. Thanks @shuchaobi for delivering it, and @CUSEAS for uploading it.

RT @Happylemon56775: Excited to share what I worked on during my time at Meta. - We introduce a Triton-accelerated Transformer with *2-sim….

RT @AleksandraFaust: Join my team at @genesistxai ! 🧬 We're forging AI foundation models to unlock groundbreaking therapies for patients wi….

RT @setlur_amrith: Since R1 there has been a lot of chatter 💬 on post-training LLMs with RL. Is RL only sharpening the distribution over co….

RT @MiniMax__AI: Day 1/5 of #MiniMaxWeek: We’re open-sourcing MiniMax-M1, our latest LLM — setting new standards in long-context reasoning.….

RT @setlur_amrith: Introducing e3 🔥 Best <2B model on math 💪.Are LLMs implementing algos ⚒️ OR is thinking an illusion 🎩.? Is RL only sharp….

RT @jacobandreas: 👉 New preprint on a new family of Transformer-type models whose depth scales logarithmically with sequence length. Enable….

RT @_arohan_: Last day today @AIatMeta, reflecting on last several months, and wanted to highlight few things I enjoyed working with:. Bui….

RT @KhurramJaved_96: Learning to play Atari from pixels from scratch in 30 minutes, all locally on an Apple Watch!

Slides here for my CVPR talk: @anoopcherian will probably know about the recording

RT @scaling01: A few more observations after replicating the Tower of Hanoi game with their exact prompts:. - You need AT LEAST 2^N - 1 mov….

RT @InfiniAILab: 🥳 Happy to share our new work –  Kinetics: Rethinking Test-Time Scaling Laws. 🤔How to effectively build a powerful reasoni….

Good take -- it's a good benchmark to develop better training algorithms / inference time scaling, which you can validate on other domains. Random / incorrect rewards won't work on this one . Main gotcha is to not overfit to just ARC- like puzzles.

Giving my first ever invited talk at @CVPR , during the multimodal reasoning workshop:. The Bitter Lesson for RL: Verification as the Key to Reasoning LLMs. This talk is inspired by the two classic essays from Rich Sutton:

RT @KMohan2006: a great video by @jbhuang0604 explaining kl divergence and its computation

RT @charlesfornlp: So many works talking about entropy, but what is the **mechanism** of entropy in RL for LLMs? 🤔. Our work gives a princi….