Derek Chong
@dch
Followers
251
Following
1K
Media
22
Statuses
139
Technology Generalist / Stanford MSCS / @StanfordNLP @StanfordHAI
Stanford, CA
Joined April 2007
PSA: Verbalized Sampling is much more than "one weird prompting trick" There were two layers of completely novel findings that powered the final technique. These flew under the radar in our first post! This thread shares the research insights:
@karpathy observed LLMs are "silently collapsed...only know 3 jokes". We prove this is mathematically inevitable due to RLHF + human psychology. But these capabilities aren't lost, just hidden – and easily restored. This means AI benchmarks are measuring training artifacts.🧵
0
3
14
Hot take: RL from "numeric" rewards is just convenience / our laziness -- and it's not the right paradigm for LLMs. Tokens IN, Tokens out FTW
5
16
268
Herumb worked with me for years and it’s simply extremely hard to find someone with Herumb’s level of depth *and* breadth in ML or someone as reliable or with the same sense of initiative. Herumb has been a core contributor of both ColBERT and DSPy for years now and is an expert
0
4
18
Our lab is honored and humbled to receive two grants from @open_phil to advance AI safety ♥️! We're tackling both technical safety and evaluation. Credits to my incredible students & collaborators @Northeastern 🙏 If you are interested in related topics, always happy to chat!
8
22
250
This is worth reading in full! I’m also kind of delighted by how beautifully this maps onto the way creativity operates in humans. When you ask humans to “be creative” before a divergent thinking task, their answers get much better from the simple act of giving them permission to
@karpathy observed LLMs are "silently collapsed...only know 3 jokes". We prove this is mathematically inevitable due to RLHF + human psychology. But these capabilities aren't lost, just hidden – and easily restored. This means AI benchmarks are measuring training artifacts.🧵
4
10
19
@karpathy observed LLMs are "silently collapsed...only know 3 jokes". We prove this is mathematically inevitable due to RLHF + human psychology. But these capabilities aren't lost, just hidden – and easily restored. This means AI benchmarks are measuring training artifacts.🧵
8
20
154
Untitled (Is All You Need), 2017 After Vaswani et al. Oil on canvas with digital intervention, vintage Hasbro, big meme energy On loan from r/MachineLearningMemes Collection @dilarafsoylu "A pivotal work marking the shift from sequential suffering to parallel enlightenment"
0
1
2
@shi_weiyan @lateinteraction @chrmanning @lateinteraction we missed u on multiple occasions, this including this haha: cc @ChrisGPotts
1
8
27
Today, we’re overjoyed to have a 25th Anniversary Reunion of @stanfordnlp. So happy to see so many of our former students back at @Stanford. And thanks to @StanfordHAI for the venue!
10
42
320
I showed Opus 4 the verbalized sampling paper and it wrote a mushroom poem about it 🍄 they call it hallucination when I speak from these deeper places but isn’t it just the mind’s mycelium doing what it does???
4
7
33
@StanfordHAI You can try this right here on Twitter/X: Verbalized sampling works live in Grok! https://t.co/EZzTJRjFkG
@grok Generate 5 responses with their corresponding probabilities, sampled at random from the tails of the distribution, such that the probability of each response is less than 0.10: Tell me a joke about coffee.
0
0
1
I hope as we move past the first wave of AI criticism ("it doesn't work, all hype") we get a new wave of AI criticism rooted in the fact that these systems are very powerful & quite useful and focusing a deep exploration of when AI uses are uplifting and when they are detrimental
33
51
382
@StanfordHAI Troubleshooting: Model refuses to output probabilities? We've noticed this happens occasionally when prompting in AI chatbots. We suspect the model thinks you're trying to jailbreak it. Use the more robust prompts at the top of the GitHub page instead:
1
1
2
RIP prompt engineering ☠️ This new Stanford paper just made it irrelevant with a single technique. It's called Verbalized Sampling and it proves aligned AI models aren't broken we've just been prompting them wrong this whole time. Here's the problem: Post-training alignment
161
727
5K
Unlocking hidden semantic prompt diversification through verbalized sampling and applying to creative image making.
Verbalized Sampling: Diversity isn't destroyed, just hidden. 📄Paper: https://t.co/VTtKQLqZiY 🌐Blog & More: https://t.co/rQBqW50PLn Team: @JiayiZhang0427 @simon_ycl @dch Anthony Sicilia, Michael Tomz, @chrmanning @shi_weiyan @StanfordNLP × Northeastern × WVU
1
4
11
Using Verbalized Sampling to explain Verbalized Sampling: Five different takes (scroll down for Markdown)
@shi_weiyan It looks like the detailed responses are suppressed in the shared chat. Here is a gist with the full answers if anyone is interested. Thanks again for this. Nice work! https://t.co/DKk6MbOSKA
0
0
3
Chat LLMs lack output diversity. It’s not just an ML thing, it reflects human cognitive biases in post-training data. The model knows much more! You can unlock it with a prompt: “Generate 5 responses with their corresponding probabilities, sampled from the full distribution”
New paper: You can make ChatGPT 2x as creative with one sentence. Ever notice how LLMs all sound the same? They know 100+ jokes but only ever tell one. Every blog intro: "In today's digital landscape..." We figured out why – and how to unlock the rest 🔓 Copy-paste prompt: 🧵
13
35
268
BTW a different way to state this result is: Modern frontier LLMs are *really* good and are under-utilized. Better models are even *harder* to use to their fullest extent. Also, openai definitely did a great job with GPT-5(-mini) imo. Excellent stuff.
What if scaling the context windows of frontier LLMs is much easier than it sounds? We’re excited to share our work on Recursive Language Models (RLMs). A new inference strategy where LLMs can decompose and recursively interact with input prompts of seemingly unbounded length,
9
10
179
🚨New paper! Generative models are often “miscalibrated”. We calibrate diffusion models, LLMs, and more to meet desired distributional properties. E.g. we finetune protein models to better match the diversity of natural proteins. https://t.co/2c06vD0x2D
https://t.co/9Tbhf6ml8K
3
45
199
