Jessica Hullman
@JessicaHullman
Followers
8K
Following
8K
Media
145
Statuses
4K
Ginni Rometty Prof @NorthwesternCS | Fellow @IPRatNU | AI beliefs uncertainty decisions metascience | Somewhere betw theory & practice | Blogger @statmodeling
Chicago, IL
Joined June 2013
Explainable AI has long frustrated me by lacking a clear theory of what explanations should do. Improve use of a model for what? How? Given a task what's max effect explanation can have? It's complicated bc most methods are functions of features & prediction but not true state 1/
5
10
69
Sycophancy. Groupthink. Test-time Inverse Scaling. What if one unsupervised metric could detect all these reasoning failures? ๐คฏ ๐ฉตPresenting our NeurIPS'25: Martingale Score๐ฉต We propose a statistical test for "Belief Entrenchment" in LLMs - no labels required. Link to paper:
1
3
10
A 'pragmatic intepretability' turn sounds a lot like our argument/framework for evaluating explanation methods--Time to replace task-agnostic fortunetelling w/concrete decision problem specs + theoretic & empirical evidence of expected performance boost https://t.co/z9XREAdjnt
arxiv.org
Modern methods for explainable machine learning are designed to describe how models map inputs to outputs--without deep consideration of how these explanations will be used in practice. This paper...
The GDM mechanistic interpretability team has pivoted to a new approach: pragmatic interpretability Our post details how we now do research, why now is the time to pivot, why we expect this way to have more impact and why we think other interp researchers should follow suit
0
1
13
Science without interpretation or ability to reproduce results. Good luck with that!
On @arxiv, we are receiving more and more submissions with short sections and many bulleted lists. I'm curious what my fellow researchers think about this style of research paper. Should this become our standard practice, or does this style omit too many details?
2
0
14
No. If this came out in the pre-AI era, almost no one would notice it. People largely donโt care about visual art for its own sake. The reason this is getting so much attention has almost nothing to do with the work itself.
Apparently this video has all of X in a frenzy. If it had come out before the AI era, people would be fawning over it as great art, but now they are so clicker trained that any mention of AI sends them into a verbiage frenzy and they anoint anything AI related as slop.
5
7
287
Deeply disappointed in my institution
Northwestern University has agreed to pay $75 million as a part of a deal with the Trump administration to restore hundreds of millions in federal funding https://t.co/rZGdjYCVoi
0
0
10
He has a jokey way of not letting people off the hook for misconduct. Eg, this made me laugh, from a stats conversation earlier this week: โActually, that reminds me of something my advisor Don Rubin used to say, when he wasnโt emailing Jeff Epstein, about how the likelihood โฆโ
0
0
2
Thanksgiving psychoanalysis of Epstein: some ppl are really good at creating elite seeming intellectual vibe networks, even if the takes one gets access to are mostly vacuous. (I mean, just look at AI lately) I love Gelmanโs refusal to let these things go. They really get to him
Larry Summers, Ken Starr, Jeffrey Epstein, and everyone else https://t.co/ACZpAWjw71
1
2
7
New Pew report confirms whatโs seemed obvious to me for awhile: X is not a place one goes to interact w/women โNever in the history of modern social media has one gender so decisively abandoned a platformโฆ X male-female imbalance is less extreme only than late-2010s Redditโ 1/2
1
7
22
Some thoughts on empirical distributions of z-scores https://t.co/TCOLMOs6Wy
0
4
23
Trying to imagine a world in which I willingly put my name on papers I've barely read. I really can't. Even a few sloppy sentencesn in a paper I'm on sticks with me. When PIs stop believing the devil is in the details, some aspect of research conscience has been lost
@thegautamkamath I think itโs fine as long as you support the team and provide intellectual input/guidance, which is vague and do not have a clear criteria. Many full professor has over 50 papers from their team. Itโs totally reasonable to put their names in the paper.
2
1
42
๐จ New working paper! How well do people predict the results of studies? @sdellavi and I leverage data from the first 100 studies to have been posted on the SSPP, containing 1,482 key questions, on whichย over 50,000 forecasts were placed. Some surprising results below.... ๐งต๐
6
65
217
Hello colleagues and friends! โจPlease help repost and share โจ: I am officially on the job market for ๐๐ฒ๐ป๐๐ฟ๐ฒ ๐๐ฟ๐ฎ๐ฐ๐ธ ๐ฎ๐๐๐ถ๐๐๐ฎ๐ป๐ ๐ฝ๐ฟ๐ผ๐ณ๐ฒ๐๐๐ผ๐ฟ and ๐ถ๐ป๐ฑ๐๐๐๐ฟ๐ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฟ๐ผ๐น๐ฒ๐ where I can apply my research to real-world AI safety challenges!!
2
35
100
The grade inflation in economics job market recommendation letters is getting out of hand. Stars and superstars and megastars. I predict that, by 2027, at least one candidate will be described as the black hole at the center of our galaxy.
20
19
471
This is a very insightful thread
Econ seminar culture is built on the assumption that the audience knows something that the speaker doesn't, and that the speaker values that information. A very important thing the audience knows and the speaker doesn't: Is the speaker making any sense at all? If nobody has any
1
3
35
ICE showed up to kidnap lawn workers and nannies today in my sleepy Chicago North Shore neighborhood. So disgusted
0
0
1
We don't fully understand the preferences human feedback encodes, so training on it can be risky. We propose a method to automatically discover these preferences! We identify unsafe, contradictory, and subjective preferences, and improve model safety, eval, and personalization.
๐ฃNEW PAPER! What's In My Human Feedback? (WIMHF) ๐ฆ Human feedback can induce unexpected/harmful changes to LLMs, like overconfidence or sycophancy. How can we forecast these behaviors ahead of time? Using SAEs, WIMHF automatically extracts these signals from preference data.
2
11
90
Our lab has produced multiple R1 tenure-track faculty ๐, won lots of paper awards ๐ & developed widely used techniques for uncertainty quantification & communication Apply to Northwestern Phd in CS https://t.co/5l7ATDIt4i or Technology & Social Behavior https://t.co/L8fypqClWl
0
0
1
๐ง โ๏ธ Interested in decision theory+cogsci meets AI? Want to create methods to rigorously design/evaluate human-AI workflows? I'm recruiting PhDs to work on ๐ฏ Stat foundations of multiagent collab ๐ซ๏ธ Uncertainty & meta-cognition ๐ Interpretability ๐ฌ LLMs in behavioral sci
1
3
14
Coming off a week-long JID-induced energy surge, I wonder how productive I'd be if I just followed the tour
0
0
5