Bored of linear recurrent memories (e.g., linear attention) and want a scalable, nonlinear alternative?. Our new paper “Test-Time Training Done Right” propose LaCT (Large Chunk Test-Time Training) — a highly efficient, massively scalable nonlinear memory with:. 💡 Pure PyTorch

I feel we need both. compression and sparsity are orthogonal, sometimes even the opposite.

RT @Haoyu_Xiong_: Your bimanual manipulators might need a Robot Neck 🤖🦒. Introducing Vision in Action: Learning Active Perception from Huma….

RT @Kai__He: 🚀 Introducing UniRelight, a general-purpose relighting framework powered by video diffusion models. 🌟UniRelight jointly model….

RT @Bw_Li1024: "Generalization means being able to solve problems that the system hasn't been prepared for.". Our latest work in #RSS2025 c….

Just arrived at Nashville for CVPR! Looking forward to chat on any topics!.

Thanks Songlin and Xinyu for hosting. Here is the recording and slides.

Happening in 5 min.

RT @xunhuang1995: Real-time video generation is finally real — without sacrificing quality. Introducing Self-Forcing, a new paradigm for t….

RT @SonglinYang4: Check out log-linear attention—our latest approach to overcoming the fundamental limitation of RNNs’ constant state size,….

RT @baifeng_shi: Finally! We just released the models and code for PS3 & VILA-HD, a vision encoder **pre-trained at 4K resolution** and the….

RT @HanGuo97: We know Attention and its linear-time variants, such as linear attention and State Space Models. But what lies in between?. I….

Intelligence needs long-context memories! We hope this work would inspire and accelerate future research in this field. 🙏 Huge shoutout to my amazing co-authors and collaborators @HaoTan5 , @Sai__Bi , @YicongHong , @KaiZhang9546 , @fujun_luan, @SonglinYang4 , Kalyan Sunkavalli,

📌Autoregressive Video Generation:. We test LaCT up to scale:  distilling a 14B param video diffusion transformer (WAN-T2V) into an AR video diffusion by replacing full attention with LaCT + SWA. (Generated videos in this tweets all comes from this model.). 8/9

📌 Language Models:. Compared to linear memory models like GLA & DeltaNet, LaCT delivers:. 🔟 5-10× larger nonlinear state.⏱ Comparable training wall-clock time.📉 Similar or better loss per token — especially at the last 2K tokens in a sequence.🔍 Similar or better retrieval

📌 Novel View Synthesis aims to render images of a static scene from previously unseen viewpoints given a set of input images. LaCT handles up to 1M tokens, outperforming 3D Gaussian Splatting with up to 128 × 960×536 input images (patch-size as 8x8 -> 1M tokens) on DL3DV

We test LaCT on 3 diverse tasks:. 🖼️ Novel View Synthesis (image sets).📝 Language Modeling (1D sequences).🎥 Video Diffusion (sequence of images). Let’s look at each ⬇️. 5/9

We do the opposite!🧠 Update fast weights using extremely huge chunks (2048–1M tokens).This simple idea has profound implications:. 🚀 Parallelism & compute intensity → 10× FLOPs utilization.🦣: Scaling of State size → up to 40% of model params in our exp. 🛠️ Simplicity → no

📚: TTT (Sun et al.) is a new way to design more powerful recurrent models. It propose to adapt a model’s fast weights during inference to store in-context info or learn in-context. It opens a vast design space for new RNNs!. But prior TTT methods suffers from low GPU

The core idea behind LaCT (Large-Chunk Test-Time Training) is simple:. 1. Use extremely large online chunk sizes (2K–1M tokens) for ttt to ensure high GPU utilization. 2. Use window attention for local memory, and test-time training (TTT) for non-local memory!. 2/9