🚀 Thrilled to share our new work, "RLoop: A Self-Improving Framework for Reinforcement Learning"! https://t.co/E2IUxgYR3t

Excited to share our research on scaling looped language models! We explore how next-generation foundation models can scale latent reasoning and more efficiently leverage parameters for knowledge manipulation!

Thank @YiyouSun for adding our work on Knapsack RL to this excellent collection. I strongly believe that focusing on the hard-tier problems—where traditional RLVR pipelines fail—is crucial for advancing our understanding and methodologies. This living repository is a vital

Thanks to AK for reposting our work! It's exciting to see how the knapsack formulation reshapes the view of system efficiency to unlock exploration and scale RL.

[8/n] If you found this work interesting, please upvote our paper! 🙏 📄 Paper:

[7/n] The budget distribution in action 📊 Our knapsack allocation creates a smart distribution: Most tasks: 1-10 rollouts (they don't need more!) Hard tasks: Up to 93 rollouts 🔥 💡 The key insight: This is a computational free lunch! → Same total compute as uniform

[6/n] Why it works: Better gradient efficiency 📈 Knapsack-GRPO maintains 70-80% effective gradient ratio vs. 50% for uniform GRPO throughout training That's +20-40% more non-zero gradients!💡 What gets solved: - Hard tasks: 549 → 306 ✅ (GRPO wastes compute here) -

[5/n] Results: Consistent wins on math benchmarks 📈 10000+ GPU hours across 4 models × 6 math tasks: Knapsack-GRPO vs. uniform GRPO: ✅ +2-4 avg points across all settings ✅ Up to +9 points on individual benchmarks ✅ 23/24 wins 💡 Key insight: Better exploration

[4/n] Our solution: Knapsack-based exploration 🎒 We formalize adaptive budget allocation as a constrained optimization: max Σ Value(N_i, p_i) s.t. Σ N_i = N_total 💡 The knapsack analogy: Each prompt with an exploration budget = an item Rollouts N_i = weight (cost) Learning

[3/n] The exploration-computation dilemma ⚖️ Hard tasks need many rollouts to explore successfully. But naive solution = scale up N uniformly? ❌ Computationally prohibitive! 📊 Example: To match our adaptive budget (93 rollouts for hard tasks), uniform allocation would need:

[2/n] Theoretical Understanding: We prove how many trials are needed to get non-zero gradients (Theorem 1) Both easy (p→1) and hard (p→0) tasks need many rollouts to get contrastive outcomes for GRPO For extremely hard tasks such as p=0.01, we would require a large value of

[1/n] Real Issue: Wasted Gradients in GRPO 🚨 In GRPO, if all rollouts have the same reward: ✅ Easy tasks → all positive → 0 gradient ❌ Hard tasks → all negative → 0 gradient → The policy gets no learning signal. Empirically (blue curve 👇), effective gradients drop

🚀 Excited to share our work at Bytedance Seed! Knapsack RL: Unlocking Exploration of LLMs via Budget Allocation 🎒 Exploration in LLM training is crucial but expensive. Uniform rollout allocation is wasteful: ✅ Easy tasks → always solved → 0 gradient ❌ Hard tasks →

[1/n] Introducing TreePO🌲, a new RL framework for LLMs! It slashes sampling costs while boosting reasoning capabilities. Daily Paper:

Is text-only information enough for LLM/VLM Web Agents? 🤔 Clearly not. 🙅‍♂️ The modern web is a rich tapestry of text, images 🖼️, and videos 🎥. To truly assist us, agents need to understand it all. That's why we built MM-BrowseComp. 🌐 We're introducing MM-BrowseComp 🚀, a new

We built FutureX, the world’s first live benchmark for real future prediction — politics, economy, culture, sports, etc. Among 23 AI agents, #Grok4 ranked #1 🏆 Elon didn’t lie. @elonmusk your model sees further 🚀🍀 LeaderBoard: https://t.co/fwck0NROHZ

🚀 Our #ACL2025 work INTP unveils how Preference Alignment improves diverse TTS models (AR, Flow-matching, and Masked Generative Model)! Unlock the secrets: ➡️ Customized Post-Training. ➡️ Human-Guided Unlearning. ➡️ Heterogeneous preference pairs to avoid reward hacking.

Amazing work by @RidgerZhu ，more resources to investigate the mechanism behind the hybrid linear attention. Resources: Paper: https://t.co/QEKocLZBEE Huggingface CKPT Link:

🚀 Thrilled to announce that our paper "SCRIT: Self-Evolving LLM Critique without Human or Stronger Models" was accepted to #COLM2025! We enable LLMs to self-improve critique abilities — zero human annotations, zero stronger models needed! 🔄✨ Looking forward to meeting

We’re excited to share our new paper “CoRT: Code-integrated Reasoning within Thinking”! 🤖 A post-training framework that teaches Large Reasoning Models (LRMs) to better leverage Code Interpreters for enhanced mathematical reasoning. 🔍 Key Highlights: Strategic hint