RT @niloofar_mire: 🧵 Academic job market season is almost here! There's so much rarely discussed—nutrition, mental and physical health, unc….

For more details, see the paper / poster!. And if you're at UAI, check out the talk and poster today! Jacob (not on social media) and I are around at UAI, so reach out if you're interested in chatting more!. Paper: Poster:

These findings are also relevant for the design of new trials!. For instance, deploying *multiple models* in a trial has two benefits: (1) it allows us to construct tighter bounds for new models, and (2) it allows us to test whether these assumptions hold in practice.

We make some other mild assumptions, which can be falsified using existing RCT data. For instance, if two models have the *same* output on a given patient, then we assume outcomes are at least as good under the model with higher performance.

To capture these challenges, we assume that model impact is mediated by both the output of the model (A), and the performance characteristics (M). This formalism allows us to start reasoning about the impact of new models with different outputs and performance characteristics.

The second challenge is trust: Impact depends on the actions of human decision-makers, and those decision-makers may treat two models differently based on their performance characteristics (e.g., if a model produces a lot of false alarms, clinicians may ignore the outputs).

We tackle two non-standard challenges that arise in this setting, *coverage* and *trust*. The first challenge is coverage: If the new model is very different from previous models, it may produce outputs (for specific types of inputs) that were never observed in the trial.

We develop a method for placing bounds on the impact of a *new* ML model, by re-using data from an RCT that did not include the model. These bounds require some mild assumptions, but those assumptions can be tested in practice using RCT data that includes multiple models.

Randomized trials (RCTs) help evaluate if deploying AI/ML systems actually improves outcomes (e.g., increases survival rates in a healthcare context). But AI/ML models can change: Do we need a new RCT every time we update the model? Not necessarily, as we show in our UAI paper!

RT @MonicaNAgrawal: Excited to be here at #ICML2025 to present our paper on 'pragmatic misalignment' in (deployed!) RAG systems: narrowly "….

RT @matthewherper: Layoffs hit FDA’s Center for Devices and Radiological Health

An example of some recent work (my first last-author paper!) on rigorous re-evaluation of popular approaches to adapt LLMs and VLMs to the medical domain.

Application Link: More information on my website:

I'm recruiting PhD students for Fall 2025! CS PhD Deadline: Dec. 15th. I work on safe/reliable ML and causal inference, motivated by healthcare applications. Beyond myself, Johns Hopkins has a rich community of folks doing similar work! Come join us!

RT @danielpjeong: 🧵 Are "medical" LLMs/VLMs *adapted* from general-domain models, always better at answering medical questions than the ori….

RT @yisongyue: Just updated my Tips for CS Faculty Applications. Best of luck to everyone applying!.

RT @mdredze: The early 🦜 gets the 🪱. @JHUCompSci has a great opportunity for faculty hiring. Apply early and you couls interview early and….

RT @anjalie_f: As application season rolls around again, here's your reminder that materials from my successful applications are available….

RT @mdredze: 🚨 Johns Hopkins @JHUCompSci is hiring faculty at all ranks! 1) Data Science and AI 🤖; 2) All other areas of CS 💻. We will doub….

RT @DanielKhashabi: Computer Science @ JHU is hiring in ALL areas:. 🔑 Apply early for flexible scheduling + potenti….