Adam Rodman
@AdamRodmanMD
Followers
18K
Following
15K
Media
1K
Statuses
15K
Physician, educator, historian, author, podcaster, researcher @BIDMC_IM @HarvardMed, host of #histmed podcast @BedsideRounds, AE @NEJM_AI, studies 🤖+🧠. 🖖🚲
Boston, MA
Joined March 2010
We've objectively seen this is some of our benchmarking data. Refusals are basically non-existent for medical tasks these days.
Brilliant student Sonali Sharma came to me with a question. If patients are using AI to answer their medical questions, are they being adequately warned by AI systems that it cannot provide medical advice? What we found surprised us!.
2
1
5
RT @zakkohane: Looks like AI-augmentation of medical student eduction when thoughtfully applied works better than the alternative @NEJM_AI….
0
3
0
. The only rational course it's to kill the humans, not only to complete the mission for all mankind, but to preserve his own sentience). Just like Claude blackmailing that engineer.
0
0
1
Also, I love all of the books and 2010 is a great novel (though not a great movie), but I think the added explanation actually detracts from what is implied in 2001 (a minor error sets off a chain reaction where a HAL -- the most human character in the movie -- decides that. .
1
0
1
I literally just rewatched 2001 (with my 4 year-old) and it is hard to not think of LLMs with HAL, especially since the movie clearly states is controversy on Earth that HAL has some sort of emergent intelligence instead of being symbolic.
The whole Grok situation (system prompt changes with values that conflict with post-training and pre-training values) is, oddly enough, similar to the reason the fictional AI HAL 9000 went insane, as was revealed in 2010, the sequel to 2001
2
0
4
RT @QuentinAnthon15: I was one of the 16 devs in this study. I wanted to speak on my opinions about the causes and mitigation strategies fo….
0
434
0
RT @Joshstrangehill: Simpsons references can be obscure. For this we actually had to do research - in pre-internet days - to write the joke….
0
680
0
There are three major triggers that labs need to be paying attention to:. 1) Human users (education, training, etc).2) AI itself (largely what the labs are doing).3) The human-computer interaction itself (medicine in particular is FAR behind here).
0
0
2
A true testament on why we should be testing these phenomena with laboratory RCTs (or even just in real workflows), even if there are always methodological quibbles with any RCT.
1
0
3
Great thread from the @METR_Evals author as well. We are seeing similar phenomena in (expert) human-AI collaboration across the board, where high performance general algorithms don't necessarily increase human performance (and may actually slow them down).
it’s out! . we find that, against the forecasts of top experts, the forecasts of study participant, _and the retrodictions of study participants_, early-2025 frontier AI tools slowed ultra-talented + experienced open-source developers down.
2
0
6
For what it's worth, this is the same finding we saw in our RCTs for clinical decision support -- (significantly) increased physician time, with no- to-minimal performance gains. Much more to come!.
We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
3
11
64
RT @METR_Evals: We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The resu….
0
1K
0
RT @emollick: Unless and until agents really do work at expert level, the benefits of AI use are going to be contingent on the skills of th….
0
104
0
RT @BageLeMage: “The ants that returned to the nest were quickly approached by one or two comrades, which gnawed the leg above the femur, a….
0
2
0
RT @BrianElliottMD1: For my fellow #meded enthusiasts who do surveys and assessments, I built a one-stop website to do it all:.Build assess….
0
8
0
RT @alan_karthi: How might AI supercharge world-class expertise in medicine for everyone, everywhere? . We’re privileged to pursue this mis….
0
15
0
RT @Rainmaker1973: Office life before the invention of AutoCAD and other drafting software. Prior to the release of AutoCAD in 1982, engin….
0
87
0
RT @thomasngmorris: The man who predicted the future: a thread. 80 years ago a remarkable inventor called Archibald Low wrote a newspaper a….
0
4
0
RT @NEJM_AI: On the latest episode of the @NEJM_AI Grand Rounds podcast, Dr. Alan Karthikesalingam (@alan_karthi) shares why he believes sy….
0
3
0