Drew Prinster @ ICML
@DrewPrinster
Followers
332
Following
1K
Media
44
Statuses
218
Trustworthy AI/ML in healthcare & high-stakes apps | My job is (mostly) error bars 🫡 (eg, conformal prediction) | CS PhD at Johns Hopkins. Prev at Yale. he/him
Joined December 2019
Paper 🧵! w/ @samuel_stanton_ @anqi_liu33 @suchisaria #ICML2024 We study 2 Qs:.1) Can AI uncertainty quant. via #ConformalPrediction extend to any data distribution?.2) Are there practical CP algorithms for AI/ML agents 🤖w/ feedback-loop data shifts?.1/
4
30
103
RT @DrewPrinster: Where's Waldo? 🧐. Maybe you'll find him on our poster, but you can definitely find me at #ICML2025!.- Invited Talk: Tue (….
0
7
0
@xinghan0 @anqi_liu33 @suchisaria Would love to chat with people thinking about AI safety monitoring, reliability, sequential testing, regulation, conformal prediction, & more, feel free to reach out! :).
0
1
2
Where's Waldo? 🧐. Maybe you'll find him on our poster, but you can definitely find me at #ICML2025!.- Invited Talk: Tue (7/15) Multi-Agent Systems in Ambient Settings Social (~7:45pm).- Poster: Thu (7/17) morning (11-1:30pm), East Exhibit Hall, E-1912.
AI monitoring is key to responsible deployment. Our #ICML2025 paper develops approaches for 3 main goals:. 1) *Adapting* to mild data shifts.2) *Quickly Detecting* harmful shifts.3) *Diagnosing* cause of degradation. 🧵w/ @xinghan0 @anqi_liu33 @suchisaria.
1
7
22
RT @BeepulBharti: I’m at #ICML2025 this week presenting my work on multiaccuracy and multicalibration with proxy sensitive attributes. If y….
0
6
0
RT @anqi_liu33: I am not going to make this ICML but @DrewPrinster will be there! Check out our recent work on online AI monitoring under a….
0
3
0
RT @TimothyDSnyder: Best read as a reasoned call to action, not a question. And we will need vivid terms for where we are going — oligarchy….
nytimes.com
And how exactly can we tell whether America has crossed the line?
0
698
0
0
0
0
@xinghan0 @anqi_liu33 @suchisaria WATCH opens up many opportunities for future work in AI safety monitoring: Eg, adaptive monitoring algorithms for other data-generating settings, extensions to monitoring generative models, LLMs, AI agents, and more!. Stay tuned for updates, & reach out if want to chat!.7/.
2
0
2
@xinghan0 @anqi_liu33 @suchisaria Takeaway 3 (Root-Cause Analysis): Beyond catching harmful shifts, monitoring should inform recovery. WATCH helps find the cause of degradation, ie by diagnosing between covariate shifts in the inputs X vs concept shifts in conditional output Y|X relation, to inform retraining. 6/.
1
0
2
@xinghan0 @anqi_liu33 @suchisaria Takeaway 2 (Fast Detection): Empirically, WATCH quickly catches harmful shifts (that degrade safety or utility of AI outputs): WATCH tends to be much more efficient than directly tracking loss metrics (& similar to standard conformal martingale baselines that it generalizes). 5/.
1
0
1
@xinghan0 @anqi_liu33 @suchisaria Takeaway 1 (Adaptation): Prior monitoring methods do sequential hypothesis testing (eg, to detect changes from IID/exchangeability), but many often raise unneeded alarms even to benign shifts. Our methods adapt online to mild shifts to maintain safety & utility! 4/
1
0
2
@xinghan0 @anqi_liu33 @suchisaria …via methods based on weighted #ConformalPrediction (we construct novel martingales), w/ false-alarm control for continual (anytime-valid) & scheduled (set time horizon) settings. Intuitively, we monitor the safety (coverage) & utility (sharpness) of an AI’s confidence sets. 3/
1
0
1
@xinghan0 @anqi_liu33 @suchisaria In real-world #AI deployments, you need to prep for the worst: Unexpected data shifts or black swan events (eg COVID-19 outbreak, new LLM jailbreaks) can harm performance. So, post-deployment system monitoring is crucial. Our WATCH approach addresses drawbacks of prior work….2/
1
0
3
AI monitoring is key to responsible deployment. Our #ICML2025 paper develops approaches for 3 main goals:. 1) *Adapting* to mild data shifts.2) *Quickly Detecting* harmful shifts.3) *Diagnosing* cause of degradation. 🧵w/ @xinghan0 @anqi_liu33 @suchisaria.
2
12
33
RT @suchisaria: 🚨With an awesome group of collaborators from health system & federal agencies, @Jean_J_Feng and I are embarking on an ambit….
0
10
0
RT @JHUCompSci: For #WorldHealthDay, a new study by Hopkins researchers including @suchisaria, @DrewPrinster, & more finds that doctors’ di….
cs.jhu.edu
A new study by Hopkins researchers finds that doctors’ diagnostic performance and trust in AI advice depends on how the AI assistant explains itself.
0
4
0
@radiology_rsna @suchisaria @PaulYiMD In sum: How AI explains its advice impacts doctors’ diagnostic performance & trust in AI, even if they don’t know it. Developers & clinical users: Keep this in mind!. Many Qs for future work…. Eg, can we dynamically select explanation types to optimize human-AI teaming? 👀. 9/9.
1
0
0
@radiology_rsna @suchisaria @PaulYiMD Takeaway 4: Doctors trust local AI explanations more than global, *regardless of if AI is correct.*. - For correct AI: Explains why local explanations improve diagnostic performance. - For incorrect AI: Local may worsen overreliance on AI errors--this needs further study!. 8/
1
0
0
@radiology_rsna @suchisaria @PaulYiMD Takeaway 2: Local AI explanations are more efficient than global: Doctors agree/disagree more quickly. Takeaway 3: Doctors may not realize how AI explanations impact their diagnostic performance! (AI explanation types did not affect whether doctors viewed AI as useful.). 7/
1
0
0