David Stutz
@davidstutz92
Followers
3,448
Following
1,299
Media
43
Statuses
403
Research scientist @DeepMind working on robust and safe AI, previously @maxplanckpress , views my own.
London
Joined January 2022
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
wonwoo
• 395781 Tweets
jeonghan
• 359680 Tweets
Mbappé
• 307248 Tweets
ANIL PIN WORLD DOMINATION
• 303673 Tweets
#TheLoyalPinPilot
• 268917 Tweets
Ucrania
• 110900 Tweets
Official MV
• 83722 Tweets
ZEGNASS25 x GEMINI FOURTH
• 79984 Tweets
Mahalima Rocks AgusanDelSur
• 70930 Tweets
रेल मंत्री
• 67714 Tweets
BILLKIN BA GUCCI IN MILAN
• 58721 Tweets
#CDTVライブライブ
• 53945 Tweets
#INI_LOUD
• 49050 Tweets
音楽の日
• 42271 Tweets
JxW LAST NIGHT OUT NOW
• 35918 Tweets
CNP x PHUWIN
• 22871 Tweets
Romania
• 21815 Tweets
酒クズアフタヌーンティー
• 18471 Tweets
サブウェイ
• 15886 Tweets
ARMY ONLY
• 14261 Tweets
Pinned Tweet
The source code for our work on evaluating and calibrating with uncertain ground truth has finally made it to GitHub - here is a thread on what’s included 🧵:
1
26
180
During PhD interviews, I feel that many prospective students have a biased view on what day-to-day work actually looks like. So I went back and checked what I actually spent time on during my PhD, and summarized it in numbers in this blog post:
13
117
871
Now that my visa is approved, I am excited to share that I will join
@DeepMind
as research scientist in the robust and verified AI team in April. Looking forward to this next chapter!
20
6
548
I was asked repeatedly for my PhD thesis and defense LaTeX templates, so I put them on GitHub. Links in thread🧵. Here are some impressions:
7
57
477
I am delighted to finally share my entire PhD thesis on robustness and uncertainty in deep learning
@maxplanckpress
@SIC_Saar
@Saar_Uni
:
11
47
426
Student researcher positions at
@GoogleDeepMind
are now open for applications until Dec 15 – see our careers webpage. Also a good opportunity to re-share my article of how I prepared for my internship back in 2019:
5
76
426
Following my PhD at
@Saar_Uni
,
@SIC_Saar
and
@maxplanckpress
, I am incredibly excited to share that I joined
@DeepMind
as a research scientist in April. I will continue to work towards robust and safe AI as part of
@DeepMind
's mission, see thread.
11
9
274
Scripts for preparing ArXiv submissions belong to these things that every researcher writes but are shared rarely. So, I decided to open source my Python scripts for cleaning and “flattening” LaTeX papers in order to put them on ArXiv:
8
46
245
I joined
@GoogleDeepMind
full-time nearly 2 years ago. And roughly one year ago, around
@NeurIPS
2022, things started changing at an incredible speed. As we are heading into 2024, I wanted to share some reflections:
4
20
245
A topic I regularly talk about with doctoral students is how to organize a PhD. It can be overwhelming to keep track of all the papers, research ideas, reviews, different collaborations, and so on. So I took some notes on how I organized my PhD:
6
46
231
We finally released the data from our work on evaluating and calibrating AI models with uncertain ground truth. This is an excellent and realistic new benchmark for machine learning with annotator disagreement - a short thread 🧵:
1
37
220
Last week I finally defended my PhD at the Max Planck Institute for Informatics
@maxplanckpress
and
@Saar_Uni
@SIC_Saar
! Got a great hat with lots of adversarial example and
@DeepMind
references from my great colleagues:
14
5
177
@DeepMind
internship applications are now open! As a research scientist and former research scientist intern I can highly recommend the internship for all current PhD students. My experience + interview prep in the thread below.
Internship applications are now open! This year we have opportunities across various teams and offices 🌎🌍
Apply today via and learn more about the experience below ⬇️
#DeepMindInterns
33
125
437
1
17
171
My take on the utility of distribution-free uncertainty estimation techniques, especially conformal prediction, in practice:
Can be seen as a response to
@beenwrekt
’s articles on the use of prediction intervals.
9
28
162
I always wanted to start a series of articles about my PhD research on adversarial examples and confidence-calibrated adversarial training. As a first step, I created several code examples for illustrating these concepts and put them on GitHub:
1
33
152
Now that my PhD thesis is online, I also want to share my PhD defense slides:
Here are some general and content-specific things I learned on the way 🧵:
2
28
147
Happy to have submitted my
#PhD
thesis at the
@Saar_Uni
,
@SIC_Saar
and the Max Planck Institute for Informatics,
@maxplanckpress
last week! Looking forward to the defense later this year.
12
2
141
Excited to share that we open-sourced our
@iclr_conf
2022 work on conformal training. Find the repository on GitHub and more details in the thread:
1
23
134
Surprisingly, I had several conversations at
@NeurIPSConf
on differentiable sorting. So I thought it might be worth to re-share our code for conformal training which includes a (jittable) differentiable sorter implemented in Jax:
1
16
114
If you are as excited as me about conformal prediction after Emmanuel Candès’ keynote at
#NeurIPS2022
or
@ChristophMolnar
email course, and you are using Jax, check out our repository with jittable+differentiable conformal predictors for classification:
0
22
109
Since I joined
@jaygshah22
on his podcast, I have had many more people ask me about “academia vs. industry” – how and why I decided to go to industry. Finally, I found the time to write down some of my thoughts in this blog article:
3
21
99
Happy to share our latest work using conformal abstention to improve factuality of LLMs by avoiding hallucinations:
A short thread 🧵:
1
15
86
Excited to announce Med-Gemini, demonstrating a new SOTA on MedQA, multimodal and long-context abilities -
I particularly want to highlight our full relabeling of MedQA, revealing that 7.4% of questions are unfit for evaluation. A short thread:
1
16
79
9 pages are too much to learn about conformal training? Check out this 4-page short paper I wrote for the ICML workshop on distribution-free uncertainty quantification:
2
13
64
Delighted to share that I received the DAGM MVTec dissertation award 2023 at
@gcpr2023
hosted by
@UniHeidelberg
this year! Thanks to my advisors Bernt Schiele and Matthias Hein and everybody else that contributed to my PhD. Personally, this is a particularly nice milestone …
12
3
63
Delighted that our work on Monte Carlo conformal prediction has been accepted at
@TmlrPub
. This was a great collaboration of
@GoogleDeepMind
+
@GoogleHealth
to enable conformal prediction for problems where even experts disagree on the ground truth. 🧵
Conformal prediction under ambiguous ground truth
David Stutz, Abhijit Guha Roy, Tatiana Matejovicova et al..
Action editor: Fredrik Johansson.
#conformal
#prediction
#uncertainty
0
6
19
1
7
63
Here is our latest work on conformal calibration with ambiguous ground truth: We show how to perform conformal calibration in settings where it is difficult to obtain crisp ground truth labels from disagreeing annotations. More in the thread 🧵.
1
8
62
I’ll be attending
@NeurIPSConf
in New Orleans. If you are around and want to have a chat, let me know via DM here or on LinkedIn. Here are some recent projects + some topics I am interested in: 🧵
3
3
58
Because it was unclear how "inaccurate" MedQA actually is, we relabeled the entire dataset with the help of clinicians. Turns out that there are several questions that are ill-posed or contain label errors - see Fig. 4b in the paper: .
Overall this is a very impressive work. Some of the techniques like uncertainty-guided search could have some broader applications. This paper also nicely demonstrates the potential applications of long context understanding for medical applications.
Of course this work clearly
2
3
34
1
8
58
It's
#ICLR2022
! Join me tomorrow at 10.30am BST / 11.30 am CEST / 2.30am PT / 5.30 ET to learn more about "learning optimal conformal classifiers". Feel free to reach out if you want to learn more but the time does not work for you or check out our poster - links in thread:
3
6
45
Looking forward to talking about our recent paper on conformal prediction with ambiguous ground truth:
Tomorrow, Tue. Feb 20, 2:00-2:50 PT. Deep Mind's
@DavidStutz92
presents to
@_AhmedmAlaa
and the Alaa lab on quantifying uncertainty with predictive algorithms when ground truths are ambiguous. Mission Hall room 1401 at UCSF or via Zoom at this link:
0
3
7
1
5
45
Final version of our
@iclr_conf
2022 paper proposing conformal training to learn deep models for conformal prediction now on ArXiv:
with
@DjDvij
@TaylanCemgilML
@ArnaudDoucet1
1/2
2
5
43
Based on a comment here on Twitter, I just updated my blog article summarizing my PhD in numbers to also shed some light on how many papers I read - and how thoroughly:
2
4
39
Thanks
@_ahmedmalaa
for the invitation. For anyone interested in learning more about conformal prediction with uncertain ground truth, check out the video below and well as the corresponding paper:
How can we apply conformal prediction when the ground-truth labels are ambiguous? Exciting new talk by
@davidstutz92
on accounting for annotators' disagreements in predictive inference!
Link:
1
8
66
0
1
38
First, the pace in AI is incredible. And it started in a time of turmoil in the tech industry. I found this combination both incredibly exciting and quite intimidating. The latter affected my motivation heavily at times ...
1
3
33
Very proud that I can finally talk about this work at
@GoogleDeepMind
trying to contribute to more responsible use of generative AI. Some of my thoughts in the thread 🧵.
We’re excited to launch 𝗦𝘆𝗻𝘁𝗵𝗜𝗗 today with
@GoogleCloud
: a digital tool to watermark and identify AI-generated images. 🖼️
It will be available on Imagen, one of
@Google
’s latest text-to-image models.
Here’s how it works. 🧵
#GoogleCloudNext
49
210
727
1
2
33
I didn’t find a perfect recipe for it but noticed that regularly talking to people around me helped immensely. It is time consuming but helped me to ignore the noise, focus on the right projects and be more motivated.
1
1
32
I haven’t made up my mind about this program yet – but it sparked quite some thoughts on how admissions, careers, publishing in AI evolved over the last few years. And I love the discussion around this happening here on X. So, here are some of my thoughts:
NeurIPS 2024 will have a track for papers from high schoolers.
79
91
598
0
3
32
The camera ready version of our
@eccvconf
2022 paper on improving robustness by enhancing weak sub-networks is now on ArXiv: - work led by
@YongGuo2219
, with Bernt Schiele
@maxplanckpress
.
1
4
29
Had the pleasure to give two talks about our work on conformal prediction with uncertain ground truth last Tue (see thread). Slides are now available on my webpage:
1
4
29
It was a pleasure chatting with Letitia at the
@HLForum
2023 about my PhD research on adversarial robustness. Listen to the full interview on her YouTube channel
@AICoffeeBreak
:
🎙️ Interview with
@davidstutz92
from Google DeepMind at
#HLF23
. We spoke about adversarial attacks and defenses for neural networks. What are hypotheses for their existence? Watch the video to find out! 👇
🎬
1
19
38
1
13
28
If you are considering to apply, check out my blog article on preparing for research internship interviews:
1
0
25
That includes me. Tried it the first time last year for our Monte Carlo conformal prediction paper () and would favor it over most conferences I submitted to before. The same also holds for reviewing!
A large fraction of submitters think that their experience with submitting to TMLR, especially reviews, are better than NeurIPS/ICML/ICLR, and almost all think it's no worse. 6/n
1
1
11
1
4
26
To kick off my blog article series on adversarial examples and adversarial training, I just published the first three articles that lay some of the foundations: monitoring training and good clean performance - 2.56% test error on CIFAR10. Start here:
1
9
24
Second, this also implied rapidly changing or progressing research projects and more focus on products. I had to learn how to wrap up ongoing projects and find out what projects are important to work on moving forward.
1
1
24
Great collaboration on conformal credal sets led by
@AlirezaJVNMRDI
. This is a great follow-up on our work on conformal prediction with uncertain ground truth, but directly conformalizing categorical distributions to form credal sets with coverage guarantee:
1/5🎉Our paper "Conformalized Credal Set Predictors" is now on ArXiv! Together with
@davidstutz92
&
@eyke_hu
, we introduced a novel approach based on conformal prediction to construct credal sets. Check it out! 📚🚀
#ConformalPrediction
#MachineLearning
1
8
26
0
1
22
I meant to write a blog article about conformal prediction and adversarial robustness but decided to instead just have a short thread 🧵 here:
1
3
23
Excited to be one of the 200 selected participants of this year’s
@HLForum
.
Congratulations to the 200 exceptional young researchers selected for this year’s 10th Heidelberg Laureate Forum! We look forward to an exciting week where promising young researchers will meet some of the brightest minds of mathematics and computer science!
#HLF23
1
6
40
2
0
22
I will be at
@NeurIPSConf
next week - finally a conference in person again. Looking forward to meeting people and talking research. So feel free to get in touch if you want to chat!
7
0
22
New pre-print by my master student Nils Walter on the role of batch normalization in learning (adversarially) robust features from fragile or random ones:
3
2
22
Really excited about attending
@HLForum
next week. Make sure to follow
@CZ_Stiftung
to get an impression of what's happening throughout the week - starting with registration tomorrow afternoon!
Heidelberg Laureate Forum starts this weekend and we're funding participation of 30 young scientists at
@HLForum
. One of them is
@davidstutz92
. He is reporting live from Heidelberg on this
#CZS
account.
#takeover
#HLF23
0
0
10
1
1
22
Last week I attended
@HLForum
for the second time. Throughout the week I took over
@CZ_Stiftung
to share impressions, but also want to do a longer thread to share what the HLF is, what we did and why I recommend young researchers to apply! A longer meta-thread ... 🧵
#CZS
#HLF23
1
1
22
This is often ignored in many papers on adversarial examples; it's great that
@goodfellow_ian
explicitly highlighted this.
I’d also like to congratulate
@biggiobattista
for independently discovering adversarial examples in approximately simultaneous work
1
8
97
0
1
21
The fifth article in my adversarial robustness series introduces confidence-calibrated adversarial training (CCAT) – an AT variant that generalizes robustness to various attacks and avoids the robustness-accuracy trade-off. Details and PyTorch code here:
0
3
20
We added text and video to SynthID and also have a really cool explainer video of how it works - 🧵
SynthID will now expand to 2️⃣ new modalities: text and video.
While this tool isn’t a silver bullet for identifying AI-generated content, it’s an important building block for helping millions of people understand the provenance of synthetic content. →
10
31
177
1
2
21
Can only recommend having a presence on the web and sharing knowledge in blog form - it's quick and does not have to be as polished as papers. Plus, alongside sharing code, it was super useful for my career, too.
10 years ago to the day, I published my first ML-related blog post:
My blogging has been very sporadic over the years, but sharing what I've learnt has been very rewarding, and probably a pretty good career move as well😁 I highly recommend it!
3
16
216
0
0
20
New pre-print by my colleague Yong Guo on improving corruption and adversarial robustness by identifying and improving particularly weak sub-networks during training using minimal computational overhead. See
2
2
18
With adversarial robustness getting more important in the context of LLMs, I wanted to share some great work by
@mxlsch
on certified robustness and adversarial training I was involved in last year:
1
0
19
Thanks again for having me
@OmarRivasplata
@stats_UCL
, it was a pleasure to present our work. For those that were unable to attend, check out this recording of an earlier talk on conformal training:
Many thanks David Stutz
@davidstutz92
for your talk earlier today (DELTA group at
@stats_UCL
). Was super interesting to hear about your work on learning optimal conformal classifiers. Great to have you and we look forward to having you again soon!
0
1
15
1
5
19
Recently, I saw quite a bit of discussion about how to choose PhD programs – maybe due to application deadlines for many programs coming up. Here is an updated version of an article I wrote in early 2018 after having started my PhD - more in thread:
1
6
18
Wrapping up our work on uncertain ground truth mainly meant coordinating people across teams while collaborators are already moving on. This was a great leadership opportunity and it’s great to see our work out there:
The source code for our work on evaluating and calibrating with uncertain ground truth has finally made it to GitHub - here is a thread on what’s included 🧵:
1
26
180
1
1
18
If you, like me, cannot wait for it to start, check out my internship work on conformal training, to be presented
@iclr_conf
2022:
1
0
18
If you want to hear about my PhD research and how I got into AI, check out the first part of my conversation with
@jaygshah22
. Thanks for having me on the podcast!
Part 1 of my fun chat w/
@davidstutz92
, research scientist
@DeepMind
on adversarial robustness & generalization in
#AI
, performance trade-offs, energy-efficient deep neural network (DNN) accelerators, and working at DeepMind.
(Part2, coming 🔜)
0
0
6
0
2
18
In the fourth article in my adversarial robustness series, I share PyTorch code to obtain a strong adversarial training baseline with 47.9% robust test error against AutoAttack - an ensemble of black- and white-box attack:
0
4
17
Thanks again for having me
@OmarRivasplata
. Great opportunity to promote adoption of conformal prediction. As before, slides at .
📢 The 6⃣th lecture of my Statistical ML module (
@Stats_UCL
) was last afternoon. Covered the basics of probabilistic models. This time our guest speaker David Stutz (
@davidstutz92
,
@GoogleDeepMind
) ✨ told us about conformal prediction 👌
Look out for the 7⃣th lecture!
0
0
12
1
2
17
The seventh article in my adversarial robustness series evaluates robustness of adversarial training against adversarial out-of-distribution examples, so-called distal adversarial examples. Article and code:
1
3
17
With AI generated content detection becoming more important and such detectors being easily fooled by adversarial attacks, I thought I would revisit my work on confidence-calibrated adversarial training with bigger models and new attacks:
0
1
16
Deciding what project to work on next was also challenging as it required to figure out what will be important going forward. Luckily our team got to be in charge of SynthID. This was a great large-scale project in the area of responsible AI:
Very proud that I can finally talk about this work at
@GoogleDeepMind
trying to contribute to more responsible use of generative AI. Some of my thoughts in the thread 🧵.
1
2
33
1
0
15
Very happy to finally share that our work on adversarial and random bit error robustness of quantized neural networks was accepted for publication at TPAMI - with one reviewer rating the paper as award quality! 1/4
2
0
14
Reading about all the progress in conformal prediction made me think of many ways to use and improve our conformal training. Here is a short blog post summarizing some of these ideas and I hope somebody picks them up and works on them:
0
4
13
Or check out this podcast with
@jaygshah22
for more tips and some impressions working at
@GoogleDeepMind
:
1
2
14
Looking forward to being a guest speaker in
@OmarRivasplata
’s course on Statistical Machine Learning next week where I will give a short introduction to conformal prediction.
📢 The 5⃣th lecture of my Statistical ML module (
@Stats_UCL
) was last afternoon. Covered generalisation, following Ch.6 📖 of "Patterns, Predictions, and Actions." The guest speaker was our very own Purvasha Chakravarti ✨
Look out for the 6⃣th lecture!
0
1
5
0
1
14
Great to be listed as outstanding reviewer for
@CVPR
the third time in a row (2020-2022). Feels good to see my reviewing work being appreciated!
0
0
14
Also wanted to share slides (in 🧵) for a second talk I gave last week on evaluating with uncertain ground truth in health at the PRECISE Center
@CIS_Penn
.
1
0
13
Regarding papers, the reality is that 80% of my submissions were rejected on the first try - for conferences - or needed major revisions - for journals. Research is an iterative process and reviewer feedback can often be discouraging and helpful at the same time.
1
0
12
Over the last weeks I made an effort to update the projects on my webpage, adding many smaller projects that are already on GitHub. I want to highlight some of them in this thread 🧵 with the hope they are useful for some people here on Twitter.
1
0
12
Piggy-backing on this, our earlier code release of conformal training also includes several conformal predictors implemented in Jax, fully jittable and even differentiable:
The source code for our work on evaluating and calibrating with uncertain ground truth has finally made it to GitHub - here is a thread on what’s included 🧵:
1
26
180
0
2
12
However, SynthID required different engineering and communication skills. For one, we put an incredible effort into (robust) evaluation – magnitudes more than what I was used to during my PhD. This required a lot of tooling, thinking about reproducibility and documentation ...
1
0
10
I invested some time today updating my webpage – you can now find a list of all my talks with slides and recordings here:
1
1
12
Publicly, people see the number of papers I published, maybe some talk and posters or my blog entries. But this is just a fraction of the work that I actually did – and I think this holds true for most PhD students. So let’s have a look behind the curtains:
1
0
11
One of the
@GoogleDeepMind
student researcher projects in our team:
Student researcher position applications are open at Google Deepmind!
I'm hosting a SR in the intersection of bias and generative models. If you're an interested PhD student please reach out!
…
0
21
48
0
1
12
@deliprao
@swamiviv1
did some great work on conformal prediction for uncertainty estimates on embeddings:
1
1
10
@y0b1byte
Thought about this quite a bit recently, see this blog post:
Most important imo: industry favors slightly different activities than academia; if these align with how you like to work, that's great. Good to be aware of these differences beforehand.
0
2
11
Finally, I put more emphasis on the community, being a GDM Scholarship mentor, being more approachable for students and others that reach out and checking out some of the AI related events in London by
@londonai
,
@weights_biases
and
@StabilityAI
:
We're hosting a MASSIVE meetup in London 🇬🇧 with
@StabilityAI
on Thursday, April 20th
Join our CEO
@l2k
and
@EMostaque
CEO of Stability AI for a night of talks on LLMs, Q&As, and fun with ML engineers!
We have limited seats so register here:
5
11
129
1
0
11
Had a great panel discussion about digital and AI tools for research at the tbh conference organized by
@germanscholars
with
@letiepi
,
@d4br4
and
@postacademia
last Friday. Here are some take-aways 🧵:
😮 To be honest, this is your last chance to join our virtual conference starting on Thursday afternoon (December 7)!
✔️ But don't worry, we have a few spots left for you!
Register here and secure yourself one of the last spots: ➡️
#tbh
0
2
9
1
2
11
And that’s only the tip of the iceberg, check out my blog article to get some more statistics, like code commits or emails written/received:
1
0
10
Thanks for having me
@OmarRivasplata
- looking forward to the talk!
Our next DELTA talk
@stats_UCL
will be on Monday 27 June at 3pm UK time, when
@davidstutz92
will tell us about *learning optimal conformal classifiers*
Details 👇
0
2
7
0
2
11
Great start of
#NeurIPS
today, catching up with people and interesting tutorials and workshops. I like the open food courts in contrast to last year, but somewhat miss the boating show ...
0
0
10
It’s great to see that our superpixel benchmark – based on my bachelor thesis in 2014 – is still actively being used.
@COmpMathematics
just contributed Docker images to ease installation:
0
1
7
I can highly recommend applying to the HLF. I attended this year and in 2019 and it was an amazing experience both times. Find some impressions + part of my application material from this year here:
Connecting laureates of mathematics and computer science with the next generation – that’s the Heidelberg Laureate Forum (HLF).
If you want to attend the 11th HLF as a young researcher, you can apply for this unique event until February 9, 2024 here:
0
3
6
0
0
9
Minor update to our
@iclr_conf
paper on learning conformal predictors end-to-end with deep networks, making the used loss more explicit. ArXiv and ICLR versions are now the same.
Learning Optimal Conformal Classifiers. (arXiv:2110.09192v3 [cs.LG] UPDATED)
0
1
6
1
1
10
In the sixth article of my adversarial robustness series, I discuss how to properly evaluate confidence-calibrated adversarial training using ensembles of adaptive attacks. Article and code:
0
1
10
The next article in my adversarial robustness series discusses a standard adversarial attack: projected gradient descent (PGD). In the article, I discuss a PyTorch implementation with several additional features such as backtracking and momentum:
0
3
10
The repository includes a Jax/Haiku implementation of conformal training and several conformal predictors for evaluation. It also allows reproducing the experiments in our
@iclr_conf
2022 paper “learning optimal conformal classifiers”:
1
0
9
It was great presenting our work on conformal prediction with uncertain ground truth at
@vanderbiltml
last Monday. You can find the slides of the talk on my webpage at .
0
0
8
For
@NeurIPSConf
, for example, I had large time slots for 1:1s and met great people throughout the week:
I’ll be attending
@NeurIPSConf
in New Orleans. If you are around and want to have a chat, let me know via DM here or on LinkedIn. Here are some recent projects + some topics I am interested in: 🧵
3
3
58
0
1
9