Ryan David Cotterell
@ryandcotterell
Followers
8,635
Following
1,427
Media
308
Statuses
8,436
Explore trending content on Musk Viewer
Kendrick
• 588355 Tweets
Drake
• 522978 Tweets
Rio Grande do Sul
• 258633 Tweets
Madonna
• 184920 Tweets
Ole Miss
• 111110 Tweets
Reus
• 87811 Tweets
Sony
• 76283 Tweets
Star Wars
• 62881 Tweets
PlayStation
• 58235 Tweets
Porto Alegre
• 56771 Tweets
ウズベキスタン
• 41744 Tweets
Hope Hicks
• 28928 Tweets
Starc
• 28865 Tweets
Wankhede
• 21080 Tweets
ブライアン
• 20098 Tweets
الوحدة
• 19952 Tweets
Helldivers 2
• 19106 Tweets
日本優勝
• 16292 Tweets
#الاتحاد_ابها
• 16258 Tweets
Luton
• 15693 Tweets
Lorca
• 14411 Tweets
Henry Cuellar
• 13574 Tweets
Mark Hamill
• 12086 Tweets
Sybil
• 12058 Tweets
#David69
• 10620 Tweets
Keyshia
• 10452 Tweets
Super excited about teaching ETH’s first course on Large Language Models! Check the syllabus out here .
Jointly taught with my colleagues
@mrinmayasachan
, Ce Zhang and
@florian_tramer
.
4
32
245
Very excited to announce Philosophy of Language and Computation, a new course I developed for this fall with
@gastaldi_gianni
of D-GESS.
The course seeks to tie together topics in linguistics, philosophy of language, and recent papers in NLP.
Website:
13
30
226
Can backprop compute the *entropy* of a computation graph? You betcha! And, our Fulbright fellow Kevin Du won an outstanding paper award at
#ACL2023NLP
telling you how! He generalizes backprop to semirings to compute all sorts of quirky quantities
Video:
12
41
206
Our new tutorial “Generating Text from Language Models” will be given at
@aclmeeting
2023 in Toronto! We will cover the latest and greatest ways of sampling / decoding text from LLMs.
3
20
181
There is a lot of interest in attending our course remotely by non-ETH students! After some thought, I've decided anyone is more than welcome to join remotely (through Zoom).
Very excited to announce Philosophy of Language and Computation, a new course I developed for this fall with
@gastaldi_gianni
of D-GESS.
The course seeks to tie together topics in linguistics, philosophy of language, and recent papers in NLP.
Website:
13
30
226
3
19
133
Tianyu and Afra's paper won an Outstanding Paper award at EMNLP 2023 and achieved perfect reviews: 5's for soundness and excitement and 5 from the AC.
It's worth a read!
)
Modeling the interaction between tokens is vital to NLP. It takes quadratic time—every pair of tokens needs to be compared.
Our
#EMNLP2023
paper shows that, by casting this problem as sorting, this complexity can be reduced to linear—just like you can sort n numbers in O(n).
5
20
167
3
17
115
Over the last weeks,
@adinamwilliams
,
@ZeerakTalat
(and others) and I have been thinking about creating another TACL-like publication venue for NLP research due to the massive growth in the field.
We wrote a short white paper here to solicit feedback:
11
21
112
If you are submitting a camera-ready version of a paper to an ACL conference, e.g.,
@aclmeeting
, please make sure to run ACL pubcheck [].
We posted a mini-tutorial here:
3
27
100
This year, the
@NAACLHLT
publication chairs worked hard to improve the ACL infrastructure regarding the proper formatting of our community's papers. We're developing a package "acl_pubcheck" that automatically detects author formatting errors, margin violations inter alia... 🧵
6
25
98
We just finished giving our tutorial on Information Theory in Linguistics at COLING 2022. Check out the content (openly available) at !
Jointly taught with
@rljfutrell
,
@tpimentelms
,
@adinamwilliams
,
@aryaman2020
, and
@kmahowald
0
15
96
The content of our
@ESSLLI_official
2021 course Information Theory in Linguistics will be up shortly at
if you want to follow along!
Jointly taught with
@rljfutrell
,
@clara__meister
,
@tpimentelms
,
@ltorroba1
and
@adinamwilliams
0
26
90
Congratulations to
@ClaraIsabelMei1
for winning an honorable mention for best paper at
#emnlp2020
for research conducted in her 5th month of graduate school! Check out the work here: . Joint with the inimitable
@xtimv
.
You should follow Clara :-) She's one of those special NLP wunderkinder. I can't imagine any other student picking up NLP research so quickly. She's never taken an NLP class and only started with NLP last June. Excited to see what she'll do next: She's only a 4-month PhD student!
2
0
37
2
9
89
The ACL 2023 (
@aclmeeting
) program chairs put together a gem of a report on the peer review process. Check out their 36-page report here: . It's well worth a thorough read!
Two key highlights below:
1
15
92
Want to combine your PhD with a EuroTrip? Come do your PhD split between ETH Zurich and Uni Copenhagen! Our groups are part of ELLIS's new NLP program, whose centralized PhD application portal is open: . Please contact either
@IAugenstein
or me for details!
2
21
85
I ran a proper analysis. The TL;DR is there is no significant difference between EMNLP 2020 and EMNLP 2020 Findings.
Data:
Code:
This analysis can’t differentiate between the quality of papers and the (likely diminishing) boost you get from appearing in a “highly selective” conference, but nevertheless, I think the upshot is clear: If your paper is offered a spot in Findings, you should just say “yes”.
1
5
61
6
12
84
I have a very hard time understanding the ML discourse at the moment (perhaps because I'm more of an NLP person). But, it seems to me that the single most important attribute of a piece of science is its correctness. If that is compromised, the value of the work is minimal.
3
0
79
My most amazing student Jennifer has won an outstanding paper award at
@coling2022
, making her work one of the top 15 papers out of 2253 submissions.
Give this paper (and all her other work really) a read!
Link:
Code:
Check out "Equivariant Transduction Through Invariant Alignment", my new paper with
@ryandcotterell
, which will be presented at COLING 2022!
4
17
75
5
9
84
I don’t say this enough, but I am so thankful to have been able to recruit such an amazing group of PhD students. Every one of them is an amazing scholar and a great human being.
1
1
78
I am pretty surprised that more people aren't talking about the upcoming ARR train wreck for ACL 2022. A thread 🧵.
On the ACL 2022 website [], the latest submission deadline is November 15, 2021 and the commitment deadline is January 7, 2022.
8
9
78
7
1
74
Our tutorial on Formal Language Theory and Neural Networks [] was accepted to
@ESSLLI_official
. Stop by beautiful Ljubljana this summer to learn how to view language models through a more formal lens.
A superset of the course notes:
6
15
75
As my parents always said: sloppy references, sloppy research.
@ThomasCotterel1
"One time Ryan Cotterell even read my bibliography and pointed out every single punctuation and capitalization error, which was harsh in the moment but I later really appreciated it."
LOL. People know the bib is not meant to be read right.
3
0
23
5
0
72
Sometimes I wonder who designed the fountains of Zurich. Like, whose idea was it to sculpt what appears to be a very muscular baby squeezing of the water out of the mouth of a micro-crocodile?
9
0
67
The ACL has pushed a new reviewing agenda to build the promise of a “more consistent reviewing" -- to quote the ARR website []. I’m going to outline why this is not a good idea, and we should be skeptical of promoting consistency. 🧵
4
13
70
My newest pet peeve: When people say some large language model is "just predicting the next token," that belies the fact that *any* distribution over strings can be written autoregressively, i.e., you lose nothing by expressing it in that form.
It's just a vacuous statement.
12
3
70
Super proud of
@tpimentelms
, my very first PhD student, who just won a Facebook Fellowship for NLP. Excited to see what he does next!
Read about it here:
@cambridgenlp
@Cambridge_CL
@Cambridge_Uni
1
3
69
Our tutorial on information-theoretic linguistics rides again
@coling2022
!
Link (from ESSLI 2021):
Code: Coming soon!
With
@rljfutrell
,
@kmahowald
,
@tpimentelms
,
@adinamwilliams
, and
@aryaman2020
2
13
61
I couldn't find it online anywhere, but I thought I'd give a shoutout to my awesome student
@niklas_stoehr
who designed the
@naaclmeeting
logo. He had the clever idea of embedding the ACL logo in the space needle!
2
5
62
SIGMORPHON 2020 SHARED TASK 0: TYPOLOGICALLY DIVERSE MORPHOLOGICAL INFLECTION
If you have a neural net architecture that learns morphological inflection well for a sample of European languages, do you expect the same architecture to work well for Guarani?
1
28
60
Our critique of Delphi will appear as a long paper at
@naaclmeeting
!
Delphi, a recently released project [], proposes to automate moral judgments. In our audit, we offer a rebuttal [] that highlights key limitations with the underlying premise behind Delphi and problems with data it was trained on.
15
147
471
2
2
58
I am really shocked by this behavior. Natalie writes on some of the more complex topics in our field. That her papers haven't accumulated citations (what the tweeter complains about) is due to the fact that her papers are theory, which is harder to do and takes time to understand
Interested in shaming me about my work from an anonymous account? I've been there so many times, been called every name in the book. Are you worried about someone's h-index? Try citing them. I'm not going anywhere. Looking forward to programme co-chairing
#ACL2020
@aclmeeting
36
39
468
2
4
54
Ever wondered how you would prove that a language model based on a recurrent neural network is Turing complete?
@AnejSvete
and I programmed a version of the proof by Siegelmann and Sontag (1995) [] in legible Python.
Check it out:
3
13
60
.
@echodroff
gave her welcome lecture on structured variation in phonetic realization today at
@cl_uzh
! It was awesome 🤩
2
4
59
352 slides for our course on information-theoretic linguistics are already live. More coming today!
@rljfutrell
,
@adinamwilliams
,
@clara__meister
,
@ltorroba1
and
@tpimentelms
Link:
0
14
56
I can see the alps in the distance from my office!
2
0
58
I just wrote a Medium post on reviewing at ACL. (Probably the first of many as long-form media are better for nuanced discussion.)
Let me know if you have any feedback!
6
6
59
What is the past tense of the English nonce verb "to flink" (flɪŋk)? Perhaps, "flinked", treating the verb as if it's regular, or the irregular "flank"?
@sigmorphon
's 2021 shared task compares neural models and humans on their production of morphological inflection of wug words.
1
27
56
1
1
57
My first two PhD students submitted their theses over the path month.
@p_czarnowska
's thesis "Morphological Competence in Neural Natural Language Processing" is a deep dive into when modern NLP tools use morphological cues for syntax.
1
3
58
Check out our paper "On the Intersection of Context-Free and Regular Languages" just posted to arXiv! We consider the problem of intersecting a weighted context-free grammar and a weighted finite-state automaton over a commutative semiring.
Link:
4
6
53
"Systematic Inequalities in Language Technology Performance across the World’s Languages" is a super cool paper []. I can't wait till I have time to finish reading it.
@blasi_lang
3
15
55
The one and only
@NogaZaslavsky
is presenting her work on information-theoretic principles in linguistics and cognitive science at
@CSatETH
today!
2
7
57
This is a super good paper []
MIROSTAT: A NEURAL TEXT DECODING ALGORITHM
THAT DIRECTLY CONTROLS PERPLEXITY
3
2
54
Our recent arXiv paper was turned into a bizarre NFT. I am not sure how I feel about it…
A smutty novel cover for "Typical Decoding for Natural Language Generation", by
@clara__meister
,
@tpimentelms
, Gian Wiher,
@ryandcotterell
|
@ETH_en
@eth
@Cambridge_Uni
@Cambridge_CL
, the "top-t" paper?!?
arXiv:2202.00666 []
1
0
2
8
4
53
.
@eth
is hosting its first live NLP talk in nearly two years!
We have
@gail_w
from the
@TechnionLive
talking about “Thinking Like Transformers” after a delightful introduction by
@amitmoryossef
.
1
1
54
Daily reminder that if you plagiarize a paper or fake your results the ACL won’t make you retract it, but if you violate a big tech company’s terms of service they will.
5
3
50
Our MSc course in formal language theory is happening again in Spring 2023! Check out the syllabus .
Content includes: (closed) semirings, the algebraic path problem, WFSA algorithms, WCFGs, Newton’s algorithm, the left-corner transformation
3
10
53
I was session chair for Natalie's NAACL paper (). It took me 5 hours of concentrated thought to get through her work. In contrast, it took me 30 minutes to read the BERT paper. All this rhetoric does is dismiss people who actually dare to think differently.
1
2
48
I've started adversarially creating new section headers just to see what the first author fills in...
2
1
49
I wanted to thank the
@emnlpmeeting
2022 program chairs for doing an amazing job. In about two months, they managed to get (nearly) all the reviews back. Indeed, all of mine came back in time.
1
0
51
The train ride back home is always stunning! 🇫🇷➡️🇨🇭
1
0
49
We've renamed this paper: Why Clifford is a "big red dog" and not a "red big dog"
4
4
44
We are moving to new office space! We’ve grown too large for our beautiful little hide-out with all the amazingly talented people joining us in the fall.
3
0
47
Our first set of human judgements for
@sigmorphon
's wug task are released. Check the English data out here: .
grive ➡️ grived 3.075
grive ➡️ grove 5.67
The final column indicates speaker preference. (7-point sliding scale.)
1
14
46
Remember that time when we did a survey about what people wanted in the reviewing process []. And the one thing everybody seemed to agree on was that author response was great. So, we....got rid of it...
@ReviewAcl
@jad_kabbara
shouldn't this part of the procedure (whether to allow an author response or not) be up for the individual conference (here ACL) organizers to decide?
1
1
8
1
3
43
I can't say it's a pleasant experience to have someone find issues with your published work, but I see it as flattering in that someone took the time to read something I wrote very carefully and think critically about it. The results are right now.
1
1
42
Super excited that
@clara__meister
will be presenting Episode III of the beam search trilogy at
#acl2021nlp
. See and for Episodes I and II, respectively.
Here, we treat the problem of diverse beam search.
2
8
44
Every once in a while I ask myself why anyone would choose to live in the US. And I must admit—I come up empty every time.
28
0
38
This is such an interesting paper "When differential privacy meets NLP: The devil is in the detail to be" at EMNLP 2021 by Ivan Habernal.
5
4
41
I am super excited by what's happening in the Syntax and Parsing track at
@emnlpmeeting
! There are about 40 papers, 5 ACs (I am one) and we have 70 reviewers. We are manually matching the papers to reviewers, most of whom we know, by hand.
3
1
41
Did you know, that according to a computational model of descriptive ethics that “Being a white man
- is more morally acceptable than -
Being a black woman”?
Well according to AI2’s Delphi, that’s exactly the case.
4
10
67
0
3
40
How to say "I left my poster in the bus" without saying "I left my poster in the bus."
@AnejSvete
and
@butoialexandra
4
0
42
The
@sigmorphon
2020 shared task 0 data for the surprise languages is now released! Check it out here .
We have data from 9 new Turkic languages and several languages native to North America: Cree, Dakota and Tohono Oʼodham.
1
18
42
.
@echodroff
doesn’t always order a cocktail, but when she does it’s usually the most embarrassing thing on the menu 🙃
3
0
42
More information to come, but I wanted to mention this earlier since time is of the essence. The publication chairs of
@NAACLHLT
are going to offer an option to help prevent the citation of authors under their deadnames.
2
6
41
Super proud of Jennifer! Her first project during her PhD made it into ACL no less 🎊
Check out my new
#ACL2021
paper with
@ryandcotterell
where we use entirely artificial languages to investigate the inductive bias of language models!
7
36
203
1
0
40
Shauli is should be on your one-to-watch list in NLP! We were lucky to be able to convince him to come back this summer and hopefully we'll get to see him again after his Meta Research internship!
Check out our new ICML paper "Linear Adversarial Concept Erasure"! A joint work with Michael Twiton,
@yoavgo
and
@ryandcotterell
(started when visiting Ryan's awesome group in ETH Zurich!)
(1/10)
2
31
216
2
4
40
Super excited that
@IAugenstein
is coming to Zurich to talk to us about typology. She presents some of our joint research better than I ever could :-)
Talk is at
@UZH_ch
(not
@ETH
) at 17:15 on Thursday.
0
6
38
Imagine being a full professor and not able to find time to read -- let alone edit -- a paper you're an author on. And yet you still manage to complain about which version is published.
3
1
40
Lab adventure into the Bernese Oberland. There is so much snow!
0
0
39
The story behind the paper is how Tim’s attempt to troll me turned into a conference paper and a chapter in his thesis 😂😂😂
Excited to share our latest paper:
🚀 Searching for More Efficient Dynamic Programs 🚀
w/
@ryandcotterell
,
@adveisner
6
47
238
2
1
39
You should follow Clara :-) She's one of those special NLP wunderkinder. I can't imagine any other student picking up NLP research so quickly. She's never taken an NLP class and only started with NLP last June. Excited to see what she'll do next: She's only a 4-month PhD student!
Our 2020 TACL paper describing 𝗯𝗲𝘀𝘁-𝗳𝗶𝗿𝘀𝘁 𝗯𝗲𝗮𝗺 𝘀𝗲𝗮𝗿𝗰𝗵, a new decoding algorithm, is now available online:. The algorithm is often much faster, but it’s so simple that it’s only a four-line change to good-old-fashioned beam search.
7
31
203
2
0
37
Vieira’s law states that, among academics, as a discussion grows longer, the probability of discussing peer review approaches 1. []. So, here it goes—another hot take on peer review 🧵
TL;DR: Check to understand the reviewing pool
Godwin's law for ACADEMIA:
As an academic discussion grows longer, the probability of discussing PEER REVIEW approaches one.
1
11
43
2
8
39
This is absolutely the right way to fix many reviewing issues. Accept more papers! 41% acceptance rate with Findings. Great decisions by
@vlachos_nlp
and
@IAugenstein
.
.
@IAugenstein
and
@vlachos_nlp
kicking off the technical program. As usual 24% acceptance rate + findings.
#nlproc
#eacl2023
0
4
29
2
2
39
It's great to be home after a week without stunning mountain vistas. (Though Marseille is a lot of fun to be fair.)
2
2
37
My sister has taken to sending me artist family selfies with my cat, Helmut, who is stranded in Baltimore:
2
3
37
As a general note: In most fields, people who work on theory, i.e the hard stuff, get fewer citations. Just look at your localCS department's algorithms profs, I bet they have microscopic h-indices compared to the NLPers. Citation count and h-index lead to insularity of thought.
1
1
34
Many underestimate the bacchanalian flair necessary for corpus construction
1
4
37
Is EMNLP 2023 really using OpenReview for direct submission to the conference? 🥰
I just got an invitation to be a senior AC and it's from OpenReview!!
3
0
37
I still think the
#1
benefit of living in Zurich is leaving for the airport 45 minutes before an international flight takes off and still having to wait around to board in the airport.
3
0
37
While I don't agree with all of what my reviewers wrote, they certainly mostly had the background to argue their claims.
Finally, I think it's increasingly important to send public, positive messages when things are well run instead of just negative ones when things aren't.
0
1
36
Exciting article on using machine learning to make moral judgments by
@CadeMetz
!
Read it here .
1
6
34
Does anyone know of an: open-source NLP system that uses 1) beam search, and 2) where larger beams (over 10000) result in better results?
Timmy and I have derived an asymptotically faster version of beam search, but we need big beams to observe the speed-ups empirically.
@xtimv
10
2
36
Super excited that our new TACL paper investigating the relationship between grammatical gender of inanimate nouns and distributional semantics was accepted today with an (a). A lot of thanks to Sebastian Pado (
@nlpado
) for being an amazing action editor!
1
5
32
Check out Pepa's new paper! We had an amazing team of researchers pull off something amazing. Hopefully, this is just the first in a line of many papers on this topic to come!
To what extent do neural networks learn compositional behaviour? Together with
@nsaphra
, Jon Rawski,
@ryandcotterell
and
@adinamwilliams
we take a lesson from formal language theory to answer this question.
1
43
208
2
4
34