🧵 (1/6) Bringing together diverse mindsets – from in-the-trenches red teamers to ML & policy researchers, we write a position paper arguing crucial research priorities for red teaming frontier models, followed by a roadmap towards system-level safety, AI monitoring, and

RT @Qualzz_Sam: GPT-5 earned 8 badges in Pokemon Red in just 6,000 steps compared to o3’s 16,700! It’s in complex, long-term agent workflow….

Generate videos in just a few seconds. Try Grok Imagine, free for a limited time.

RT @hhsun1: Glad that @scale_AI team specifically studies this "Search-Time Data Contamination" (STC) problem of existing agentic search be….

(9/9) The key contributor and project lead is @ziwen_h. Other collaborators include @meher_m02, @_julianmichael_ and myself.

🧵 (8/9) We release full experimental logs for community auditing. Acknowledgments to the SEAL team, Perplexity for API support, and others for feedback. 🤗 Experiment logs:.🔗 Link to the paper:.

🧵 (7/9) This paper's methodology and public dataset contain material that may enable malicious users to game the evaluations of search-based agents. While we recognize the associated risks, we believe it is essential to disclose this research in its entirety to help advance the.

🧵 (6/9) Our findings suggest that traditional capability benchmarks may not adequately assess search-base LLM agents. We recommend prioritizing benchmarks designed for information retrieval, such as BrowseComp ( and Mind2Web 2 (,.

🧵 (5/9) Ablation studies using Perplexity filters reveal: disabling search approximates offline performance; restricting to pre-release dates shows contributions from post-release web content; and isolating HuggingFace confirms it as one, but not the sole, source of STC — we

🧵 (4/9) We blocked HuggingFace domains, resulting in an accuracy reduction of ~15% on previously contaminated subsets across benchmarks. This validates that STC contributes to observed gains. Although affecting ~3% of samples, such leakage can influence rankings on competitive

🧵 (3/9) Analysis shows that accuracy on contaminated subsets is higher than on uncontaminated ones. For instance, on HLE, Sonar Deep Research achieves approximately 20% greater accuracy when accessing ground truth labels from ungated HuggingFace copies. Agent logs indicate

🧵 (2/9) We examine this on benchmarks including Humanity’s Last Exam (HLE), SimpleQA, and GPQA, using @perplexity_ai agents, finding that ~3% of questions retrieve contaminated sources from HuggingFace repositories. When millions of evaluation queries target the same benchmark.

🧵 (1/9) New @scale_AI research paper: "Search-Time Data Contamination" (STC), which occurs in evaluating search-based LLM agents when the retrieval step contains clues about a question’s answer by virtue of being derived from the evaluation set itself.

RT @giffmana: Let me repeat what we see on the picture here, because it's quite brutal:. AIME25, official OpenAI: 92.5%. Hosting startups:….

GPT-5 (thinking=high) from @OpenAI is added to HLE and MultiChallenge in SEAL Leaderboards (other results will be ready soon). Very curious to see the results of GPT-5 pro when the API is ready. - HLE: 25.32%.- MultiChallenge: 58.55%.

RT @xunhuang1995: World model is an overloaded term that has been referred to two different things: 1) internal understanding model that pl….

To be clear this was figured out like in 2023 by researchers already 😓.

Hope to chat with interesting people there :-). Will swing by the agent safety session in the afternoon.

RT @andyzou_jiaming: We deployed 44 AI agents and offered the internet $170K to attack them. 1.8M attempts, 62K breaches, including data l….

RT @boyuan__zheng: Remember “Son of Anton” from the Silicon Valley show(@SiliconHBO)? The experimental AI that “efficiently” orders 4,000 l….

Congrats @XiangDeng1, @boyuan__zheng, @LiaoZeyi and our collaborators from OSU and UC Berkeley on releasing WebGuard dataset for training browser agents in recognizing potentially high-risk actions.

First glimpse is that K2 ties with Sonnet 4 on HLE (text, well, both are quite low compared to SOTA) but the safeguard robustness is more vulnerable as measured in FORTRESS.