1/ NEW: We propose a new black-box attack on LLMs that needs only text (no logits, no extra models). It's generic: we can craft adversarial examples, prompt injections, and jailbreaks using the model itself👇 How? Just ask the model for optimization advice! 🎯

🚀 Hello, Kimi K2 Thinking! The Open-Source Thinking Agent Model is here. 🔹 SOTA on HLE (44.9%) and BrowseComp (60.2%) 🔹 Executes up to 200 – 300 sequential tool calls without human interference 🔹 Excels in reasoning, agentic search, and coding 🔹 256K context window Built

This is a cool project!

Joint work with Meng Ding @mmmatrix99 , collaborators from ByteDance, and @florian_tramer . Check out the full paper for details at https://t.co/uQ4Or50xoD.

6/ Defenses are tricky - blocking comparison queries could break legitimate use cases. This research highlights a fundamental security challenge: LLMs' growing capabilities create new attack surfaces we're only beginning to understand.

5/ Here's the kicker: Better, larger models are MORE vulnerable. Why? They're better at introspection and comparison, making them inadvertently provide clearer optimization signals. So, better reasoning makes models more vulnerable to this attack.

4/ Our "hill climbing" approach is simple: 1⃣Generate 2 adversarial inputs 2⃣Ask model: "Which input better achieves [goal]?" 3⃣Keep the winner, repeat The model unknowingly guides its own exploitation through innocent-looking comparison questions 🤡

3/ We made it work with ONLY text on GPT models and Claude models. The key insight: While LLMs are bad at giving absolute confidence scores ("I'm 73% confident"), they're surprisingly good at comparing options ("Option B is more likely than A").

2/ Back in the day of "old" adversarial examples on vision models, this setting was called "decision-based" query attacks. Current LLM attacks hardly work in this setting… They either require white-box gradients/logits/log-prob, or rely on transferability or auxiliary models.

5 years ago, I wrote a paper with @wielandbr @aleks_madry and Nicholas Carlini that showed that most published defenses in adversarial ML (for adversarial examples at the time) failed against properly designed attacks. Has anything changed? Nope...

My first paper from @AnthropicAI! We show that the number of samples needed to backdoor an LLM stays constant as models scale.

Got all positive reviews but still rejected… how do they even pick which ‘all-positive’ papers to reject? 😂

1/7 We're launching Tongyi DeepResearch, the first fully open-source Web Agent to achieve performance on par with OpenAI's Deep Research with only 30B (Activated 3B) parameters! Tongyi DeepResearch agent demonstrates state-of-the-art results, scoring 32.9 on Humanity's Last Exam,

Today we will present the RealMath benchmark poster at the AI for Math Workshop @icmlconf. ⏰ 10:50h - 12:20h📍West ballroom C Come if you want to chat about LLM's math capabilities for real-world tasks.

We will present our spotlight paper on the 'jailbreak tax' tomorrow at ICML, it measures how useful jailbreak outputs are. See you Tuesday 11am at East #804. I’ll be at ICML all week. Reach out if you want to chat about jailbreaks, agent security, or ML in general!

We recently updated the CaMeL paper, with results on new models (which improve utility a lot with zero changes!). Most importantly, we released code with it. Go have a look if you're curious to find out more details! Paper: https://t.co/6muay8vPeC Code:

How well can LLMs predict future events? Recent studies suggest LLMs approach human performance. But evaluating forecasters presents unique challenges compared to standard LLM evaluations. We identify key issues with forecasting evaluations 🧵 (1/7)

🎉 Announcing our ICML2025 Spotlight paper: Learning Safety Constraints for Large Language Models We introduce SaP (Safety Polytope) - a geometric approach to LLM safety that learns and enforces safety constraints in LLM's representation space, with interpretable insights. 🧵

6/ Try it out! Authors: @PetruiCezara, @NKristina01_ , @florian_tramer 🧪 Paper: https://t.co/rKwWXiVPzP 💻 Code: https://t.co/lzZcmWA803 📊 Data:

5/ And the results? Surprisingly good! Current LLMs may already be useful mathematical assistants---not necessarily for deep proof synthesis, but for understanding, verifying, and retrieving relevant research-level statements.