The most creative open-source team I've ever known.

Interestingly, a lot of tech reports for industrial LLMs trained with μP do not mention "μP" but "special" scaling laws, in the shade of MetaP's failure. This is quite like the AI ​​winter 20 years ago when people referred to AI as cognitive systems or computational intelligence.

🎉 Our paper "Tensor Product Attention Is All You Need" has been accepted as NeurIPS 2025 Spotlight (Top 3%)! The Camera Ready version of TPA has been publicly available on the arXiv now: https://t.co/5AJoEjl6oH ⚡️TPA is stronger and faster than GQA and MLA, and is compatible

it’s an improved version of Gated DeltaNet. enjoy ^^

@HuizhuoY @QuanquanGu 8/n We are glad to announce that our MARS-M paper is released: https://t.co/2a2Vb0G96u. It extends variance reduction framework of MARS to matrix-level optimizers, demonstrating consistent performance gains over Muon in LLM pretraining tasks. GitHub Repo:

We are so back, Long live REINFORCE! See also our RPG paper: https://t.co/BNclBdUg4Z

🥂A friendly mapping for folks reading the MiniMax‑M1 report: the awesome work CISPO can be written as a clean instantiation of RPG‑REINFORCE ( https://t.co/H8FZPFLfSL), an off-policy REINFORCE Algorithm with clipped importance sampling weight.

@HuizhuoY @QuanquanGu 7/n MARS-M and approximated version of MARS-M:

6/n Thank @HuizhuoY and @QuanquanGu for the great collaboration.

5/n Tagging some people who might be interested: @JingyuanLiu123 , @kellerjordan0 , @MatPagliardini , @_arohan_ , @ShamKakade6 , @bremen79 , @aaron_defazio , @peter_richtarik , @HazanPrinceton , @MarkSchmidtUBC , @zhiyuanli_ , @cloneofsimo , @jxbz , @HessianFree

4/n Our work combines the best of both worlds: variance reduction (MARS) and matrix-based optimization (Muon). Both have been shown to be among the most effective methods for improving AdamW in recent optimizer benchmarks by @wen_kaiyue, @tengyuma, @percyliang, @AndreiSemenov17.

3/n MARS-M empirically validates that variance reduction techniques, like MARS, can be adapted to matrix-based optimizers to achieve consistent and stable performance improvements.

2/n MARS-M enhances Muon by incorporating scaled gradient correction into momentum, which significantly reduces variance during model training. In GPT-2 experiments, MARS-M beats Muon at every step with the same hyper-params, and outperforms AdamW on training & validation loss.

1/n We introduce MARS-M, which extends our variance reduction framework, MARS, to matrix-based optimizer Muon (Moonlight). MARS-M demonstrates consistent performance gains over Muon in LLM pretraining tasks. Github Repo: https://t.co/vuubxbk3iQ

🎉Our paper, "Tensor Product Attention Is All You Need," has been accepted for a spotlight presentation at the 2025 Conference on Neural Information Processing Systems (NeurIPS 2025)! See https://t.co/5AJoEjl6oH and https://t.co/0y5bySxU4S for details!

Interesting conclusions for MARS in these papers. Glad to see that MARS performs well in the researches.

Another fantastic benchmark of optimizers. Key takeaways: 1. Variance-reduced Adam variants (e.g., MARS) achieve significant speedups over the AdamW baseline. 2. Matrix-based optimizers (e.g., Muon, SOAP) consistently outperform their scalar-based counterparts (e.g., Lion).

Why does CANADA try to prevent AI researchers from attending conferences in Cadana? I doubt whether Canada wants to develop their AI industry. Why does CANADA try to prevent ones named for maple to enter Canada? I doubt whether Canadians love maples.

Which optimizer (from 100+ optimizers for DL models) is best for training Large Language Models? 🤔 https://t.co/GDVmzzmSts

6/6 🚀Experimental Results: RPG beats GRPO/DAPO/REINFORCE++ We implement RL training experiments on math datasets with Qwen-2.5-7B-Instruct and Qwen-2.5-Math-7B, achieving more stable and better performances than baselines including GRPO, DAPO and REINFORCE++.