🌟 Excited to share our latest work on making diffusion language models (DLMs) faster than autoregressive (AR) models! ⚡ It’s been great to work on this with @caglarml 😎 Lately, DLMs are gaining traction as a promising alternative to autoregressive sequence modeling 👀 1/14 🧵

RT @iScienceLuvr: Inverse Scaling in Test-Time Compute. "We identify five distinct failure modes when models reason for longer: 1) Claude m….

RT @XiuyingWei966: If you’re interested in long-context efficiency, don’t miss our recent paper RAT—a joint effort with @anunay_yadav, Razv….

RT @caglarml: Many people still talk about coming up with alternatives to self-attention, but acknowledging the strengths of both self-atte….

RT @jdeschena: 🔥 NEW PAPER: "The Diffusion Duality". Uniform-state diffusion models for text generation emerges from an underlying continuo….

RT @OHilliges: Sadly, I am no longer a professor at ETH (@eth_en) due to very severe #longCovid and #MECFS.

RT @ssahoo_: Attending ICML ✈️Tues-Fri to present "The Diffusion Duality".🗓️Wed, July 16 @ 4:30pm.📍East Exhibition Hall A-B (E-3003). DM if….

RT @SkanderMoalla: 🚀 Big time! We can finally do LLM RL fine-tuning with rewards and leverage offline/off-policy data!. ❌ You want rewards,….

RT @XiuyingWei966: Curious about making Transformers faster on long sequences without compromising accuracy? ⚡️🧠 Meet RAT—an intermediate d….

RT @johnowhitaker: I did another video, on the paper 'The Diffusion Duality', continuing the series of me trying to understand diffusion ap….

RT @sedielem: This work uncovers a profound connection between continuous and discrete (non-absorbing) diffusion models, allowing transfer….

RT @_akhaliq: The Diffusion Duality. unlock few-step generation in discrete diffusion language models via the underlying Gaussian diffusion….

RT @SkyLi0n: Check out our recent paper on the "duality" between discrete and Gaussian diffusion. We show how you can exploit that relation….

@DrYangSong 🔗 CODE & MODELS:. 📜 Paper: 📘 Blog: 💻 Code: It was amazing to work on this with @ssahoo_, @SkyLi0n, @Guanghan__Wang, @justintchiu, @volokuleshov, onto the next 🚀.9/9 🧵.

@DrYangSong 🎯 SUMMARY. • Duo bridges continuous and discrete diffusion.• Enables faster training with curriculum learning.• Enables faster sampling by adapting consistency models to discrete spaces.8/9.

@DrYangSong Using argmax at the final step cuts this to 8 steps, with only a slight drop in unigram entropy 🤯 7/9.

@DrYangSong Discrete Consistency Distillation slashes sampling steps by orders of magnitude: after distillation, we can sample in just 16 steps and match the original generative perplexity—without lowering unigram entropy. 6/9

Importantly, the Diffusion Duality lets us adapt Consistency Models (@DrYangSong) to discrete diffusion! 🔄✨ Even though there is no PF-ODE for discrete spaces, we can use the Gaussian PF-ODE to generate distillation trajectories, then transfer them to the discrete domain! 5/9

After 1M training steps, Duo sets a new SoTA 🏅 among comparable uniform-state diffusion models, thanks to curriculum learning! 📈 Duo also outperforms AR models on 3 out of 7 zero-shot datasets 🚀 4/9

The curriculum learning strategy slashes training steps by 2x by reducing variance: early during training, instead of discrete values, we feed the neural network a low-temperature softmax over the Gaussian diffusion vector, which accelerates training 🚀 3/9

🔍 THE CONNECTION:. When you apply argmax to vectors computed through Gaussian diffusion, you get uniform-state discrete diffusion. It is quite amazing that such a simple operation connects two seemingly disconnected paradigms 🤯 2/9