(1/n) 🚀Thrill to share our LoRA-One work ( https://t.co/3MrW0e0vii) as #ICML25 𝐨𝐫𝐚𝐥 𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧, w. Fanghui @Fanghui_SgrA (Warwick) and Yudong (Madison). Oral @ West Ballroom B, 4pm at July 17th Poster @ West Exhibition Hall B2-B3 #W 905, 4:30PM at July 15th

LLMが持つ代表的な能力である文脈内学習についての論文を公開しました。本研究では、Transformerが ・最適なメタ学習法を事前学習で獲得できること ・その結果、文脈内学習として短いプロンプトからタスクへ即座に適応し、最適な回答を出力できること を示しました。 https://t.co/dA6f5qbFFI

Towards Formalizing Reinforcement Learning Theory.

- Test-time Adaptation of Tiny Recursive Models - New Paper, and the Trelis Submission Approach for the 2025 @arcprize Competition! In brief: - @jm_alexia's excellent TRM approach does not quite fit in the compute constraints of the ARC Prize competition - BUT, if you take a

New paper we're excited to get online! Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking. A totally new framework based on ~backtracking~ for using process verifiers to guide inference, w/ connections to approximate counting/sampling in theoretical CS.

This seems like a major breakthrough for AI advancement Tencent and Tsinghua introduced CALM (Continuous Autoregressive Language Models), a new approach that replaces next token prediction with continuous vector prediction, allowing the model to think in ideas instead of words.

Unbelievably detailed new paper from DeepMind on benchmarks and autograders they used on IMO Gold journey. For me main takeouts are: - autograding can achieve ~90% accuracy even on long and difficult reasoning - DeepThink is quite behind IMO Gold model on very difficult problems

New @Microsoft paper teaches LLMs to organize reasoning into concurrent subtasks for faster, more accurate answers. It shows 28% lower wait time than typical parallel thinking while also boosting math accuracy. The big deal is simple, it turns coordination into a skill the

I've found the path to AGI: https://t.co/NRvZViBVUc

What if next-token prediction wasn't a single forward pass, but a tiny optimization problem? Introducing: nanoEBM a tiny transformer that learns to think harder by doing gradient descent on its own predictions. You can start training on your Mac now - it comes < 400 lines

My paper "Understanding Softmax Attention Layers: Exact Mean-Field Analysis [...]" accepted at #NeurIPS2025 @NeurIPSConf. This was a fun way to apply some stat physics techniques I learned recently, on a non-trivial problem. Infinity stones are almost complete :) Preprint soon!

We just built and released the largest dataset for supervised fine-tuning of agentic LMs, 1.27M trajectories (~36B tokens)! Up until now, large-scale SFT for agents is rare - not for lack of data, but because of fragmentation across heterogeneous formats, tools, and interfaces.

This paper provides a geometric analysis of Principal Component Analysis (PCA) to determine what data properties define its excess risk. 🔬 The authors establish a central limit theorem for the principal subspace and derive the asymptotic distribution of its excess risk,

@jm_alexia @ritteradam Indeed, @jm_alexia @ritteradam I also find that simply increasing the number of inference steps, even when the model is trained with only 16, can substantially improve performance. (config: TRM-MLP-EMA on Sudoku1k; though the 16-step one only reached 84% instead of 87%)

🤯 Merging many finetuned LLMs into one model, effectively? Introducing Functional Dual Anchor (FDA), a new framework for model merging. 🚀 Current merging works poorly due to the underlying parameter conflicts. FDA shifts knowledge integration to the input-representation space

To establish power law behavior we need statistical tests. This paper is a nice overview of statistical methods for testing power laws "Power-Law Distributions in Empirical Data" by A Clauset, CR Shalizi, & MEJ Newman SIAM Review, 51(4), 661–703 https://t.co/jlYNEXrekT 4/5

Tiny Reasoning Language Model (trlm-135) - Technical Blogpost⚡ Three weeks ago, I shared a weekend experiment: trlm-135, a tiny language model taught to think step-by-step. The response was incredible and now, the full technical report is live:

@jasondeanlee @CL_Theory @Fanghui_SgrA (5/n) We also construct a benchmark consisting of 2,894 gold-standard DAG-MATH formatted CoT (problems from Omni-MATH) as demonstration examples for few-shot prompting, also serving as high-quality data for SFT.

@jasondeanlee @CL_Theory @Fanghui_SgrA (4/n) We propose the 𝐃𝐀𝐆-𝐌𝐀𝐓𝐇 format to reveal DAG structure of CoT. With this, we evaluate three LLM families on 3 popular math benchmarks via our new metric. We found that the test models have similar perfect reasoning ability although there are large gaps in PASS@1.

@jasondeanlee @CL_Theory @Fanghui_SgrA (3/n) In details, under DAG framework, we can extract a sampled DAG from each CoT trajectory. We call the CoT logically close if any intermediate sampled node has at least one child. We term logically closed CoT with correct final answer as 𝐏𝐞𝐫𝐟𝐞𝐜𝐭 𝐑𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠.

@jasondeanlee @CL_Theory @Fanghui_SgrA (2/n) Building on this framework, we introduce 𝐥𝐨𝐠𝐢𝐜𝐚𝐥 𝐜𝐥𝐨𝐬𝐞𝐧𝐞𝐬𝐬, a metric that quantifies how well a model’s CoT trajectory (i.e., the LLM’s output solution) adheres to the DAG structure, providing evaluation beyond classical PASS@k metrics.