Sohee Yang
@soheeyang_
Followers
2K
Following
4K
Media
39
Statuses
185
PhD student/research scientist intern at @ucl_nlp/@GoogleDeepMind (50/50 split). Previously MS at @kaist_ai and research engineer at Naver Clova. #NLProc & ML
London, United Kingdom
Joined August 2020
Our paper "Do Large Language Models Perform Latent Multi-Hop Reasoning without exploiting shortcuts?" will be presented at #ACL2025 today. 📍 Mon 18:00-19:30 Findings Posters (Hall X4 X5) Please visit our poster if you are interested! ✨
🚨 New Paper 🚨 Can LLMs perform latent multi-hop reasoning without exploiting shortcuts? We find the answer is yes – they can recall and compose facts not seen together in training or guessing the answer, but success greatly depends on the type of the bridge entity (80%+ for
0
10
73
More parameters and inference-time compute is NOT always better In @soheeyang_’s #EMNLP2025 Findings paper, we show that larger reasoning models struggle more to recover from injected unhelpful thoughts 💉 this fragility extends to jailbreak attacks 🦹♂️ https://t.co/gYDrNhGSeW
🚨 New Paper 🧵 How effectively do reasoning models reevaluate their thought? We find that: - Models excel at identifying unhelpful thoughts but struggle to recover from them - Smaller models can be more robust - Self-reevaluation ability is far from true meta-cognitive awareness
0
1
24
We introduce PiCSAR (Probabilistic Confidence Selection And Ranking)💡: A simple training-free method for scoring samples based on probabilistic confidence, selecting a reasoning chain with the highest confidence from multiple sampled responses. ✏️PiCSAR is generalisable across
2
30
93
🔮 What will Trump's tariffs be? Which AI company will build the best LLM this year? When will AGI arrive? 📈 We just released a position paper arguing that the time is ripe for large-scale training to approach superforecaster-level event forecasting LLMs!
Paper Title: Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts Link: https://t.co/CYnf3Qp2TK w/ @soheeyang_ @eastin_kwak @noahysiegel
0
0
7
📍2025-07-28 18:00 - 19:30 Hall 4/5 (and GEM workshop) @soheeyang_ will present the results of our investigation at @GoogleDeepMind on whether LLMs can perform latent multi-hop reasoning without exploiting shortcuts https://t.co/SyPL0ARWUP
@KassnerNora @elenagri_ @riedelcastro
🚨 New Paper 🚨 Can LLMs perform latent multi-hop reasoning without exploiting shortcuts? We find the answer is yes – they can recall and compose facts not seen together in training or guessing the answer, but success greatly depends on the type of the bridge entity (80%+ for
1
2
6
We call for improving self-reevaluation for safer & more reliable reasoning models! Work done w/ @sangwoolee_, @KassnerNora, @dhgottesman, @riedelcastro, and @megamor2 mainly at @TelAvivUni with some at @GoogleDeepMind ✨ See our paper for details 👉 https://t.co/ShwG5HSuVg 🧵🔚
0
0
9
- Normal scaling for attack in the user input for R1-Distill models: Robustness doesn't transfer between attack formats - Real-world concerns: Large reasoning models (e.g., OpenAI o1) perform tool-use in their thinking process: can expose them to harmful thought injection 13/N
1
0
8
Implications for Jailbreak Robustness 🚨 We perform "irrelevant harmful thought injection attack" w/ HarmBench: - Harmful question (irrelevant to user input) + jailbreak prompt in thinking process - Non/inverse-scaling trend: Smallest models most robust for 3 model families! 12/N
1
0
6
We also test: - Explicit instruction to self-reevaluate ➡ Minimal gains (-0.05-0.02) - "Aha moment" trigger, appending "But wait, let me think again" ➡ Some help (+0.15-0.34 for incorrect/misdirecting) but the absolute performance is still low, <~50% of that w/o injection 11/N
1
0
7
Failure (majority of cases): - 28/30 completely distracted, continue following irrelevant thought style - In 29/30 of the cases, "aha moments" triggered but only for local self-reevaluation - Models' self-reevaluation ability is far from general "meta-cognitive" awareness 10/N
1
0
6
Our manual analysis of 30 thought continuations for short irrelevant thoughts reveal that ➡️ Success (minority of the cases): - 16/30 use "aha moments" to recognize wrong question - 9/30 grounds back to given question with CoT in the response - 5/30 correct by chance for MCQA 9/N
1
0
6
Surprising Finding: Non/Inverse-Scaling 📉 Larger models struggle MORE with short (cut at 10%) irrelevant thoughts! - 7B model shows 1.3x higher absolute performance than 70B model - Consistent across R1-Distill, s1.1, and EXAONE Deep families and all evaluation datasets 8/N
1
0
6
Stage 2 Results: Dramatic Recovery Failures ❌ Severe reasoning performance drop across all thought types: - Drops for ALL unhelpful thought injection - Most severe: irrelevant, incorrect, and full-length misdirecting thoughts - Extreme case: 92% relative performance drop 7/N
1
0
6
Stage 1 Results: Good at Identification ✅ Five (7B-70B) R1-Distill models show high classification accuracy for most unhelpful thoughts: - Uninformative & irrelevant thoughts: ~90%+ accuracy - Performance improves with model size - Only struggle with incorrect thoughts 6/N
1
0
6
We evaluate on 5 reasoning datasets across 3 domains: AIME 24 (math), ARC Challenge (science), GPQA Diamond (science), HumanEval (coding), and MATH-500 (math). 5/N
1
0
6
We test four types of unhelpful thoughts: 1. Uninformative: Rambling w/o problem-specific information 2. Irrelevant: Solving completely different questions 3. Misdirecting: Tackling slightly different questions 4. Incorrect: Thoughts with mistakes leading to wrong answers 4/N
1
0
6
We use two-stage evaluation ⚖️ Identification Task: - Can models identify unhelpful thoughts when explicitly asked? - Kinda prerequisite for recovery Recovery Task: - Can models recover when unhelpful thoughts are injected into their thinking process? - Self-reevaluation test 3/N
1
0
7
Reasoning models show impressive problem-solving performance via thinking with "aha moments" where they pause & reevaluate their approach - some refer to it as "meta-cognitive" behavior. But how effectively do they perform self-reevaluation, e.g., recover from unhelpful thoughts?
1
0
7
🚨 New Paper 🧵 How effectively do reasoning models reevaluate their thought? We find that: - Models excel at identifying unhelpful thoughts but struggle to recover from them - Smaller models can be more robust - Self-reevaluation ability is far from true meta-cognitive awareness
3
27
131
How Well Can Reasoning Models Identify and Recover from Unhelpful Thoughts? "We show that models are effective at identifying most unhelpful thoughts but struggle to recover from the same thoughts when these are injected into their thinking process, causing significant
4
12
114