Excited to announce our #AISTATS📜on training LLMs with MXFP4! We use stoch. rounding and random Hadamard transforms (all fast on HW) to get low-variance, unbiased gradient estimates with MXFP4 GEMMs. We get a ~30% speedup over FP8 with almost no PPL gap!.

RT @yingheng_wang: ❓ Are LLMs actually problem solvers or just good at regurgitating facts?. 🚨New Benchmark Alert! We built HeuriGym to ben….

RT @ellisk_kellis: New paper: World models + Program synthesis by @topwasu.1. World modeling on-the-fly by synthesizing programs w/ 4000+ l….

RT @justachetan: I will be at #CVPR2025 presenting our work on differential operators for hybrid neural fields! Catch me at our poster:. 🗓️….

RT @simran_s_arora: Checkout CARTRIDGES, scaling cache-time compute! An alternative to ICL for settings where many different user messages….

RT @EyubogluSabri: When we put lots of text (eg a code repo) into LLM context, cost soars b/c of the KV cache’s size. What if we trained a….

RT @tri_dao: Albert and co continue to do excellent work on quantization. This time the trick is to minimize KL wrt the original model, wit….

Apparently I chose the worst day to release a paper, so ICYMI, we made a post-training quantization algorithm that outperforms even @Google's quantization-aware training recipe. We beat the prior SOTA by >30%, meaning faster and smaller models. More details in the original 🧵👇

RT @JenJSun: VideoPrism is now available at: :).

RT @togethercompute: 5/ Quantized models don't need to lose fidelity. Check out our paper and blog for details:. 📝 Paper: .

RT @austinsilveria: chipmunk is up on arxiv!. across HunyuanVideo and Flux.1-dev, 5-25% of the intermediate activation values in attention….

RT @togethercompute: 🚀 New research: YAQA — Yet Another Quantization Algorithm (yes, pronounced like yaca/jackfruit 🥭). Led by @tsengalb99,….

@chrismdesa @togethercompute *.

@chrismdesa (6/6) We also have a blog post ( with @togethercompute, who graciously provided compute resources for this project!.

(5/6) All this results in a lower KL and SOTA downstream performance across a wide range of models and quantizers. For more information, check out our (w/Zhaofeng Sun & @chrismdesa) paper ( and code (.

(4/6) YAQA’s rounding algorithm comes with nice theoretical guarantees that allow us to reason about YAQA’s behavior vs. LDLQ. In fact, we show that LDLQ is a special case of YAQA that uses a provably worse Hessian than YAQA’s.

(3/6) YAQA solves this by quantizing to directly minimize the KL to the original model. YAQA first computes near-optimal Kronecker-factored Hessian approximations for the KL in a fully-distributed way, and then uses these Hessians in a new adaptive rounding algorithm.

(2/6) Existing quantization methods like GPTQ and LDLQ (QuIP, QuIP#, QTIP) typically round to minimize the immediate activation error. However, reducing this metric does not necessarily reduce the end to end KL and produce a closer model!.

📣Introducing our latest work: Yet Another Quantization Algorithm!. YAQA directly minimizes the KL divergence to the original model during rounding, cutting it by >30% over prior PTQ methods and giving an even closer model than Google’s QAT on Gemma! 🤯. �

RT @turboderp_: I made a thing.

Apparently ExLlama3 is based off of QTIP and I just found out today?!? 🤯.