Zachary Lipton
@zacharylipton
Followers
62K
Following
22K
Media
741
Statuses
13K
Cofounder & CTO: @AbridgeHQ, Professor: CMU/@acmi_lab, Creator: @d2l_ai & https://t.co/QQt98VNLUp, Relapsing 🎷
Joined January 2009
Yesterday, I had the honor of speaking in the US Senate AI Insight Forum on privacy & liability, moderated by @SenSchumer @SenatorRounds @SenatorHeinrich & @SenToddYoung. Each panelist submitted a written statement for @SenSchumer's website. Here’s mine:.
14
31
179
Many ML practitioners have no idea where to start getting a foothold in learning theory. I certainly didn't. "Understanding Machine Learning" by Shai Shalev-Shwartz & Shai Ben-David is an absolute treasure. Beautiful hardcover & free download @
26
373
2K
I don’t know why so many people need to hear this, but if you actually want to do a PhD, don’t only apply to “top 10” schools. The cutthroat-ness of admissions falls off rapidly but the quality of faculty does not.
52
112
1K
By default, logistic regression in scikit-learn runs w L2 regularization on and defaulting to magic number C=1.0. How many millions of ML/stats/data-mining papers have been written by authors who didn't report (& honestly didn't think they were) using regularization?.
47
205
1K
pro·fes·sor — noun.***the member of a research lab with the deepest intuitions about how LaTeX \vspace, page breaks, and figure placement work.***.
22
112
1K
Well here goes. Our ICML Debates paper is live: "Troubling Trends in Machine Learning Scholarship". If anyone needs me, I'll be in witness protection. 🙄
29
367
945
So much fucking respect to @ylecun for being the one big-tech blue chip name to step up & champion open research, open source & the startup community. Unflinchingly. For remembering what made this moment possible in the first place & putting his name on the line to protect it ❤️.
7
58
957
The field is already self-correcting. Good departments/labs are clearing their eyes, caring less about paper count, seeing through the noise. Don't worry so much about the ICML deadline. Slow down, relax, try to do work you're proud of, submit when it's ready.
8
137
942
Our ***free deep learning book*** ( now has a ***modern NLP chapter draft***, complete w. the omnipresent BERT & friends. All drafted via interactive Jupyter notebooks.
We have re-organized Chapter: NLP pretraining ( and Chapter: NLP applications (, and added sections of BERT (model, data, pretraining, fine-tuning, application) and natural language inference (data, model).
8
188
876
When an entire sub-field of ML has failed (thus far) to produce any method that could actually be used, there is no "state of the art", only the "state of the game".
13
52
429
Tomorrow, we launch "The Art of the Paper", a course on the principles, mechanics, & culture of scientific writing. While core to our work, this material seldom gets a formal treatment. I'm excited, nervous, & grateful to @mldcmu for the creative freedom.
4
143
849
Thrilled to release our greatest paper yet in Nature Machine Learning: ***Multimodal Multitask Multidomain Multi-Attention for Multilabel Classification with Multiple Adversaries***
42
77
756
If you follow known fraudster & plagiarist @sirajraval, ***please take a moment to unfollow him***. Especially prominent researchers, journalists, and institutions. Don’t confer credibility on this scoundrel and thereby help to defraud his students.
38
173
766
Instead of receiving a boring paper diploma at the end of their PhDs, Google PhD graduates receive a priceless note handwritten by Jeff Dean on a napkin that says, “You have a PhD now.”
15
85
756
Type I and type II errors are right up there with “bandits” as the most obnoxious, least descriptive technical terms that have ever been adopted.
31
35
719
The dominant practice of the applied machine learnist has shifted from ad-hoc feature hacking (2000s) to ad-hoc architecture hacking (2010s) to ad-hoc pre-training hacking (2020s).
20
61
710
If you want to go into CS academia, first be a lousy jazz musician for 10 years. You will never find the pay low (even in PhD), the amount to learn onerous (even in mature subfields), the feedback cruel (even from R2), or the the rejection rate high (even at NeurIPS/ICML/COLT).
16
37
678
***In the first thrilling installment, the Superheroes of Deep Learning find themselves up against an ordinary problem. Will their extraordinary abilities carry the day?***. New on Approximately Correct! @AndrewYNg @drfeifei
22
137
672
Guys, I have a burning question. What's the best language for data science, Python or R?.
274
74
648
For as long as I've been a computer scientist, I've been convinced that I am the worst possible mathematician one can be while still hoping to do meaningful ML research. This feeling hasn't changed at all as my abilities have fluctuated.
19
31
654
Physics is searching for a *theory of everything.*. Deep learning is searching for a *theory of anything*.
25
44
645
***ML Tragic Pattern*** (1) Propose algorithm that makes no sense. (2) Ignore obvious examples that illustrate the incoherence. (3) Implement it anyway (4) Fool reviewers @ a top conference. (4) Celebrate. (5) "Discover" method has problems. (6) Write "critical" paper.
17
69
645
If I ever shared how abusive & toxic my experience of the business technologies group at Tepper was, the group would cease to exist. It still rattles me every day. I couldn’t even share my joy when I escaped from that hellhole. Too traumatic to acknowledge I ever worked there.
27
43
627
***OpenAI Trains Language Model, Mass Hysteria Ensues*** New Post on Approximately Correct digesting the code release debate and media shitstorm.
13
175
581
The double descent phenomenon is described in ~1000 papers & talks over past year. It's featured in at least 1 slide per talk @ last summer's Simons workshop on Foundations of DL. Why is this @OpenAI post getting so much attention as if it's a new discovery? Am I missing smtg?.
A surprising deep learning mystery:. Contrary to conventional wisdom, performance of unregularized CNNs, ResNets, and transformers is non-monotonic: improves, then gets worse, then improves again with increasing model size, data size, or training time.
24
69
589
BREAKING NEWS: Many of you are panicking about where to find more toilet paper. Turns out every general audience AI book from the last 5 years contains 250+ sheets. 🧻🚽.
16
64
565
When I grow up, my dream is to become the i-th (3 < i < N) author of a major @Google paper for an intern-level contribution, then set $1B dollars on fire. Later, I plan to play Call of Duty in a villa bought w secondary sales while employees pay the cost as it all implodes.
12
34
577
10x researchers. Professors, if you ever come across this rare breed of researchers, grab them. If you have a 10x researcher as part of your lab, you increase the odds of winning a Turing award significantly. OK, here is a tough question. How do you spot a 10x researcher?.
16
84
562
Here’s a puzzling fact about the mainstream ML/AI community: a bunch of ostensibly creative dreamers were handed job security, massive amounts of cash, and unprecedented creative license—and the only thing most ppl can think to do with this freedom: train bigger networks. 🤣.
How does a university based researcher keep feeling relevant in this fast changing and compute driven field?. Some good discussion here ➝
23
69
556
Anyone want to guess what deep learning hotshot drives this hot rod?
16
35
549
It’s hard to understate just how strange the state of “explainable AI” is today. For nearly all “local explanation” techniques, the only people who understand them would never use them, and the only people who use them do not understand them.
15
81
520
Overjoyed to announce that I have N papers accepted to the [name of international conference on X]! [no details about the work or coauthors follow] #conferencenameYYYY.
9
30
503
There’s been a rash of deep learning & VC bozos saying things like “with enough data, correlation is causation”. This is incorrect, damaging to the discourse, & makes us (empirical ML researchers/practitioners & tech broadly) look like idiots to the science-literate world.
17
58
490
Curious fact abt the LLM era: it's strangely difficult to distinguish the methodological innovations of elite research labs from the efforts of random bloggers.
18
27
504
New life plan: (1) launch a fledgling AI startup; (2) somehow get publicly listed with no business plan & few routes to solvency; (3) convince every hedge fund to short the shit out of my stock; (4) befriend /u/DeepFuckingValue.
10
31
490
I'm all for a "slow movement" in ML publishing (see my tweet suggesting 1-yr paper moratorium last wk). But is the right avatar for such a movement a man who ostensibly authored *92* papers last year? Let's discuss (& pls keep it constructive everyone).
36
62
493
Awesome ML blog run by @lilianweng has fantastic exposition, clear illustrations and covers a wide spectrum of topics in classic and modern ML:.
5
98
499
Overall the ML community at present does not have this problem. .
Don't let perfect be the enemy of submitted.
12
19
479
Why is ML "taking over" so many adjacent fields, stealing the spotlight & overshadowing (often-comparable) work? The rapid pace of innovation? The talent? The tooling? I'm inclined to think the major (unsung?) factor might just be the universality of ***open-access publishing***.
29
53
475
I think most of us find Einstein's diaries upsetting, notably for indulging in ugly stereotypes when discussing his travels in Asia. At the same time, I'm conflicted about the decisions to publish dead people's private diaries. Thoughts?
69
54
441
The precarious state of “interpretable deep learning” is that we should be far more scared upon hearing that a hospital or government deploys any such technique than upon hearing that they haven’t.
18
63
458
Just spent a full day shadowing radiologists reading mammography and diagnostic breast cancer scans (MR, ultrasound, etc). Huge opportunities to use ML to *help* but ***wow wow wow*** are some people underestimating the work radiologists do.
22
118
445
4
37
462
***What were the most interesting, scientifically insightful, and coherent papers that you read in 2019??*** . Rules: .1) No restriction on methodology or aim (theoretical, experimental, & applications papers welcome). 2) Do not post your own papers.
32
88
456
In most ML textbooks (& intro-y stats books), distributions fall out the sky, w/o full derivations of their precise functional forms (e.g. why the gamma function appears in the Beta). Does anyone have a great intro prob/stats books that works everything up from first principles?.
25
60
446
“If you haven't run a baseline of logistic regression, you are committing algorithmic malpractice.” –@scorbettdavies at @ucsantabarbara Responsible ML panel. YES.
5
83
445
I've seen a lot of BERT tutorials, not a single one that you could read and thereafter reproduce anything close to BERT (without consulting source code). Is there a more complete tutorial out there I'm missing?.
23
46
426
If @overleaf ever truly dies, I will quit research, move to Greece, and spend the rest of my days baking under the sun, reading sci-fi novels, & drinking away the pain.
22
23
417
Insisting that authors include a LIME/SHAP/IG/TCAV/GRADCAM saliency map in a paper shd be a disqualifying offense for reviewers. Including any such map without a powerful disclaimer (“this means nothing”) shd be a disqualifying offense for authors. Extra true for ML in healthcare.
28
51
417
People often ask if academia can keep up in research with industry given the superior resources there. Honestly, 99% of *interesting* academic work *even in DL* requires < 16 GPUs. This is maybe 1/100th the cost of an electron microscope. Resources are a bit over-emphasized.
19
66
409
Dear world (CC @businessinsider, @Hamilbug): stop saying "an AI". AI's an aspirational term, not a thing you build. What Amazon actually built is a "machine learning system", or even more plainly "predictive model". Using "an AI" grabs clicks but misleads
26
90
399
Hey everyone. Stoked to report that we've blown away recent NLP benchmarks with a new sentence embedding: "Efficient Recurrent Neural Inverse Embeddings" Idea's simple: iteratively embed & invert sentences gains thru meta-learning magic! #NLP #deeplearning.
19
56
412
***Uncontroversial opinion***: Geoff Hinton & Jurgen Schmidhuber both made remarkably creative (and prescient) contributions to deep learning. The credit assignment process is broken and biased, with pockets of work overlooked, but not necessarily due to malice of any individual.
10
28
402
Do pretraining’s big wins in NLP really involve “knowledge transfer”? Are upstream corpora even needed? *Not always!!* My students @kundan_official & @saurabh_garg67 show that self-pretraining (from scratch) often rivals “foundation model” performance.
7
82
403
Advice to a young academic in 2016: get on Twitter. Advice to a young academic in 2021: get off Twitter.
14
21
393
Absolutely delighted to join the Operations Research group (joint w. MLD) at @teppercmu. With ML now applied in the wild, we must address both *predictions* & *decisions*. This nexus is where I see my research headed & I couldn't hope for stronger colleagues on either side.
12
13
395
Dear @overleaf, if you implement a feature that allows us to search for projects by collaborator names, my PhD student @dkaushik96 will tattoo "@overleaf" on his biceps and name his first child @overleaf.
10
10
391
Don't go into a career in research for the fast cash that may come & go w. industry interest. Go into research for the freedom to wake up every day & decide that the most important thing in (your) world is to read about topic X, then do it, and then annoy people about it.
4
70
391
Super excited to announce that “Dive Into Deep Learning” now officially supports PyTorch!
Dive into Deep Learning now supports @PyTorch. The first 8 chapters are ready with more on their way. Thanks to DSG IIT Roorkee @dsg_iitr, particularly @gollum_here who adapted the code into PyTorch. More at @mli65 @smolix @zacharylipton @astonzhangAZ
9
54
386
Quarantine has made one thing clear: The real reason for conferences is to force weeklong breaks from meetings.
5
16
369
Anyone who tells you they are not jaw-on-the-floor surprised that you can have this sort of interaction with a chatbot today is lying.
12
8
375
Ali Rahimi delivered a rare critical talk at #nips2017, likening modern ML to alchemy. Examples: brittleness of SGD to implementation changes & mystical claims like "batch norm works by reducing internal co-variate shift". Full talk now on YouTube:
3
139
358
Me: [rambles while introducing an idea, apologizes].@dkaushik96: "no, keep going. this is doing wonders for my impostor syndrome". 🤣🥳😅. — The Joys of PhD Advising, 2020.
5
5
365
After a decade of carefully maintained social media silence, our dear visionary @geoffreyhinton breaks his twitter silence… to gripe about retail banking customer service.
Does HSBC UK have any ML people? HSBC will not comply with my written instructions to transfer money within the UK. Fraud detection says it must be authorized by high value transfers. High value transfers say they cannot authorize it. 7 hours on the phone so far. Help!.
10
17
370
Can someone give me tenure already so that I can write an honest book about the 2010s AI boom?.
17
13
358
I can't imagine being a woman in CS seeing this week's news. I feel shocked and devastated. Foremost for the victims. But also for all the other women who have to put their guard up higher, doubting the authenticity of the professional attention they receive. You deserve better.
3
17
348
Academia is not special because it's perfect or because it always lives up to ideal. It's easy to point to BS work and bad processes to trash the institution. But the 5% of work that is both magical & unlikely to happen anywhere else makes the entire enterprise worth it.
6
31
350
eight yrs ago i was a jr phd student over the moon to be sharing beers with the inventor of the lstm (@HochreiterSepp) at neurips in montreal w my first deep learning collaborator @davekale. hope you get to share a human moment w someone you admire this week.
3
20
285
For all the criticism that I direct towards ML, I'm grateful to belong to a community open-minded enough to laugh at itself and flexible enough to change. Having experienced more conservative, closed-minded corners of academia, I know it's a luxury not to be taken for granted.
7
10
352
Importance weighting is widely used but ***may have no effect on deep nets*** modulo choices regarding early stopping and weight decay. This "science-y" paper with my student Jonathon Byrd identifying this phenomenon was just accepted at #ICML2019 (1/2).
5
50
352
Business school professors get paid 1.5–2x CS. Asked why, university mumbles abt outside options. But CS profs have 2-5x outside earning power. Time's long nigh to re-calibrate & cut B-school pay.🔪.
33
18
358
Interviewing Elon Musk and Mark Zuckerberg for expert opinions on AI is like interviewing the president of Switzerland abt particle physics.
17
106
335
Notable: while LLMs are the singular force shaping the discipline & dominating the discourse in NLP, I saw absolutely nobody from the major LLM shops (@OpenAI, @AnthropicAI, @MosaicML) make an appearance at #acl2023.
19
26
347
Is the best Variational Autoencoder (VAE). not actually a VAE at all?!? This paper is wild. Extremely simple idea: regularized deterministic AE + ex-post density estimation. Results somehow both unsurprising—intuitive!—& shocking—how did people not know this for years?!?.
@skoularidou @_sam_sinha_ @zacharylipton but that uses adversarial learning in one variant (and MMD in another). Let me shamelessly plug RAEs The catch is that ex-post density estimation on the latent space gets you better aggregate posterior estimation and hence sample quality.
4
65
342
The perennial Q of "how to ensure the US continues to lead in AI?" is so easy to answer. Fully fund all PhDs in strong programs outright, grant visas to best students in the world, & let them stay afterwards. It would be the cheapest grand initiative in the history of government.
4
53
326
The reading list for pilot run of 10721: Philosophical Foundations of Machine Intelligence has been finalized and presenter notes from groups to date are up on the course GitHub for anyone interested in reading along:.
13
65
335
Note to all ML authors: w k-fold cross validation, you *must* have an independent test set. Especially in healthcare, I've seen many papers this year with no test set. This qualifies as a desk reject.
17
81
332
One interesting (but incomplete) measure of the state of the deep learning bubble. While PyTorch has caught up with TensorFlow, aggregate interest in DL frameworks seems to be maybe half what it was at the peak. I can imagine many interpretations, share yours.
45
43
317
Idea of the week: ***One year moratorium on papers.*** For the entire community. A whole year of thinking without sprinting/hustling/spamming towards deadlines.
20
18
327
In latest ad campaign, @IBM says solution to homelessness is [wait for it. ] AI. The virtuosos have hit a new peak for shameless cynical bullshit.
17
30
315
Biden is very likely going to win, but going forward, can we stop acting like @NateSilver538 knows more than any other semi-smart blogger with lumpy prose and an undergraduate's knowledge of statistics?.
28
16
324
Excited to announce some personal news: After 3 yrs as science advisor to @AbridgeHQ, I’m jumping in as Chief Scientific Officer. Our team of NLP/ML researchers, designers, & engineers are tackling some of the biggest pain points facing doctors & patients. More to come…
24
10
330
Before the media blitz & retweet party get out of control, this idea exists, has been published, has a name, and a clearer justification. It is called ***Counterfactually-Augmented Data*** and here's the published paper (spotlight at #ICLR2020).
7
59
318
@weatherbryan @jonkeegan Are those airplanes all in the air simultaneously? That's terrifying.
8
8
288
Help me Twitter. Best (reasonably technical) math/science podcasts?.
33
41
316
Prompts are not Lipschitz. There are no “small” changes to prompts. Seemingly minor tweaks can yield shocking jolts in model behavior. Any change in a prompt-based method requires a complete rerun of evaluation, both automatic and human. For now, this is the way.
13
40
315
This fall I'm leading "CMU 10732: Robustness & Adaptation in Shifting Environments" focusing on causally structured shift, adversarial attacks, strategic classification, online adaptation, & feedback loops in recsys. Follow our readings & lecture notes:
6
56
313
"It's me Siraj, let's talk about quantum computing." => ."It's **Maine** Siraj, let's talk about quantum **cipher**".
30
30
317
There's a handful of ML ideas that just *feel right*---perhaps due to evoking some aspect of human learning?---that keep recurring but never seem to have panned out convincingly. Here's two: (1) curriculum learning; (2) hierarchical reinforcement learning. (Dis)agree? Got others?.
30
26
310
BTW, deep learning libraries are full of this—e.g., undeclared weight decay. All kinds of things lurk that seem like good ideas using the "works better in general" argument but actually create a sloppy environment where scientists don't really know what model they are using.
By default, logistic regression in scikit-learn runs w L2 regularization on and defaulting to magic number C=1.0. How many millions of ML/stats/data-mining papers have been written by authors who didn't report (& honestly didn't think they were) using regularization?.
13
39
308
Across all political & ideological commitments, there's a movement today away from civil discourse & towards signalling, dunking, & mindless pile-ons. It's aided (or caused) by the amplification dynamics encoded in our platforms, and it's ruining us all. We can & must do better.
15
24
294
Why have a "standard route" at all? It seems that the greatest poxes facing the ML community are intellectual homogeneity & group-think. Having everyone take too-similar paths to PhD appears (to me) the very pathology implicated. Granted, as a strange-path freak, I'm biased.
13
26
300
Before you write the word "novel" 15 times in a paper, think: are you planning to work in ML for more longer than this hype cycle? Do you anticipate respectable (read *scientifically literate*) ppl reading your paper or is your audience only the VC you're presently defrauding?.
13
48
299
Amazing how many in the deep learning community can simultaneously decry the closed-minded monoculture that spurned NN research in the 90's and celebrate an even narrower monoculture in the 2020's.
3
32
288
If twitter dies, where are 70% of the world's greatest scholars going to spend 5 hours per day posting pallid platitudes?.
16
14
298
Excited to join CMU & also wistful abt leaving UCSD. They took me in for PhD 4yrs ago when I was jazz musician w/o research experience, a life-changing opportunity that I fear might become rarer as field grows more competitive & PhD programs are swamped w experienced applicants.
7
8
288
Steps to get hired as an "AI researcher" in silicon valley:.1. Download a deep learning library.
16
49
283
Is there any known case of anyone accessing “harmful capabilities” of an LLM that didn’t consist of knowledge already freely available and clearly described in documents on the open web?. Is the fear that we are basically just getting what we would already have if Google / Bing.
34
42
291
I live in Pennsylvania & so can you. PhD & MS applications to @mldcmu & @teppercmu open now! *Move to Pittsburgh, do great research, have a voice.*. If you work on fairness, healthcare, distribution shift, feedback loops, or NLP (but more than BERT) consider applying to @acmi_lab.
9
31
286