Dan Deutsch
@_danieldeutsch
Followers
613
Following
120
Media
16
Statuses
90
Research Scientist at Google Translate working on text generation evaluation
San Francisco
Joined September 2012
Excited to receive an Outstanding Paper award for this work at @emnlpmeeting! Thanks to my co-authors George Foster and @markuseful! Updated version available here:
LLM-based metrics like GEMBA predict many ties, but the way that ties should be handled in Kendall’s tau for meta-evaluating metrics has been a longstanding issue. We propose an update to the meta-evaluation methodology to handle ties.
4
9
70
RT @iseeaswell: Working on Low Resource Languages? Want to help with SMOL? join our new discord!
0
1
0
RT @markuseful: Two new datasets from Google Translate targeting high and low resource languages!. WMT24++: 46 new en->xx languages to WMT….
0
26
0
RT @iseeaswell: 😼SMOL DATA ALERT! 😼Anouncing SMOL, a professionally-translated dataset for 115 very low-resource languages! Paper: https://….
0
11
0
@shrutirij @prk_riley @esalesk @FirasTr88060642 Stephanie Winkler @BZhangGo @markuseful . #nlproc #nlp #ai.
1
0
1
This project was a highly collaborative effort with many people contributing translations, evaluations, analyses, etc., so I want to thank all of my co-authors! @ebriakou @iseeaswell @marafinkels Rebecca Galor @JurikJuraska @gezakovacs Alison Lui @RicardoRei7 @jasonriesa.
1
0
2
As a bonus, we also release screenshots of the original English URLs to support future work on multimodal translation.
1
0
2
We also find that MT systems/LLMs produce better translations than the human translators (at least according to automatic metrics), a result that should be confirmed with a follow up study (hint: future work coming soon!)
1
0
2
We collect machine translations from a variety of different MT systems/LLMs and rank them with automatic metrics. Takeaway: LLMs get the best scores across all languages! Data coming to MTME soon
1
0
3
Each language has a human-written reference and subsequent post-edit (including for 8 of the original WMT24 languages).
1
0
2
🚨New machine translation dataset alert! 🚨We expanded the language coverage of WMT24 from 9 to 55 en->xx language pairs by collecting new reference translations for 46 languages in a dataset called WMT24++. Paper: Data:
5
25
90
RT @mykocyigit: Thrilled to share our latest findings on data contamination, from my internship at @Google! We trained almost 90 Models on….
0
19
0
RT @JurikJuraska: 🚀 We have just released bfloat16 variants of all 3 MetricX-24 models, offering nearly identical performance to their floa….
0
2
0
RT @JurikJuraska: 🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now ope….
0
6
0
Super simple and effective way of significantly increasing the performance of your evaluation metric!.
LLMs are typically evaluated w/ automatic metrics on standard test sets, but metrics + test sets are developed independently. This raises a crucial question: Can we design automatic metrics specifically to excel on the test sets we prioritize? Answer: Yes!.
0
0
8
The Google Translate Research Team is looking for interns this summer! Apply here if you will graduate from a PhD program in the 2025-2026 academic year, and send me an email to let me know that you applied.
3
51
185
New application link! I am at EMNLP/WMT this week. Please come find me if you want to learn more about this role!.
Interested in doing research on Google Translate and Gemini? Good news! I’m hiring for full-time roles on the Google Translate Research Team! Apply here:
0
10
35
Please get in contact with me if you have any questions!.
0
0
4
Running large-scale experiments for building SOTA MT models.
1
0
2
Researching how to improve the quality of human evaluations of text generation.
2
0
4
