Zachary Lipton
@zacharylipton
Followers
58,981
Following
1,987
Media
734
Statuses
12,618
Professor: CMU/ @acmi_lab , CTO / CSO: @AbridgeHQ , Creator: @d2l_ai & , Relapsing 🎷
Pittsburgh, PA
Joined January 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Kendrick
• 498242 Tweets
Drake
• 467320 Tweets
Rio Grande do Sul
• 255048 Tweets
ONGSASUN LOVEBIRDS
• 247164 Tweets
Ole Miss
• 102287 Tweets
Reus
• 78091 Tweets
Sony
• 65258 Tweets
#الهلال_التعاون
• 65189 Tweets
猫の恩返し
• 45372 Tweets
ウズベキスタン
• 31690 Tweets
#MIvsKKR
• 20718 Tweets
Leclerc
• 20136 Tweets
Al Green
• 16685 Tweets
Hope Hicks
• 16213 Tweets
Odell
• 14474 Tweets
ブライアン
• 13121 Tweets
Lorca
• 10151 Tweets
Pinned Tweet
Yesterday, I had the honor of speaking in the US Senate AI Insight Forum on privacy & liability, moderated by
@SenSchumer
@SenatorRounds
@SenatorHeinrich
&
@SenToddYoung
. Each panelist submitted a written statement for
@SenSchumer
's website. Here’s mine:
10
29
166
Many ML practitioners have no idea where to start getting a foothold in learning theory. I certainly didn't. "Understanding Machine Learning" by Shai Shalev-Shwartz & Shai Ben-David is an absolute treasure. Beautiful hardcover & free download @
27
384
2K
I don’t know why so many people need to hear this, but if you actually want to do a PhD, don’t only apply to “top 10” schools. The cutthroat-ness of admissions falls off rapidly but the quality of faculty does not.
55
118
2K
By default, logistic regression in scikit-learn runs w L2 regularization on and defaulting to magic number C=1.0. How many millions of ML/stats/data-mining papers have been written by authors who didn't report (& honestly didn't think they were) using regularization?
50
215
1K
Why the frack is
@google
renting the top search result for "Joe Biden" out to ad hominem attack ads? Who is asleep at the wheel?
@sundarpichai
@WittedNote
@JeffDean
[***This space should not be for sale in an election***]
64
283
1K
pro·fes·sor — noun
***the member of a research lab with the deepest intuitions about how LaTeX \vspace, page breaks, and figure placement work.***
24
121
1K
Well here goes. Our ICML Debates paper is live: "Troubling Trends in Machine Learning Scholarship". If anyone needs me, I'll be in witness protection. 🙄
32
402
1K
So much fucking respect to
@ylecun
for being the one big-tech blue chip name to step up & champion open research, open source & the startup community. Unflinchingly. For remembering what made this moment possible in the first place & putting his name on the line to protect it ❤️
8
61
1K
The field is already self-correcting. Good departments/labs are clearing their eyes, caring less about paper count, seeing through the noise. Don't worry so much about the ICML deadline. Slow down, relax, try to do work you're proud of, submit when it's ready.
8
141
986
Our ***free deep learning book*** () now has a ***modern NLP chapter draft***, complete w. the omnipresent BERT & friends. All drafted via interactive Jupyter notebooks.
We have re-organized Chapter: NLP pretraining () and Chapter: NLP applications (), and added sections of BERT (model, data, pretraining, fine-tuning, application) and natural language inference (data, model).
3
109
389
8
198
913
When an entire sub-field of ML has failed (thus far) to produce any method that could actually be used, there is no "state of the art", only the "state of the game".
17
113
516
Tomorrow, we launch "The Art of the Paper", a course on the principles, mechanics, & culture of scientific writing. While core to our work, this material seldom gets a formal treatment. I'm excited, nervous, & grateful to
@mldcmu
for the creative freedom.
4
150
899
Thrilled to release our greatest paper yet in Nature Machine Learning: ***Multimodal Multitask Multidomain Multi-Attention for Multilabel Classification with Multiple Adversaries***
46
80
797
If you follow known fraudster & plagiarist
@sirajraval
, ***please take a moment to unfollow him***. Especially prominent researchers, journalists, and institutions. Don’t confer credibility on this scoundrel and thereby help to defraud his students.
40
181
795
Instead of receiving a boring paper diploma at the end of their PhDs, Google PhD graduates receive a priceless note handwritten by Jeff Dean on a napkin that says, “You have a PhD now.”
16
93
783
Type I and type II errors are right up there with “bandits” as the most obnoxious, least descriptive technical terms that have ever been adopted.
34
37
780
The dominant practice of the applied machine learnist has shifted from ad-hoc feature hacking (2000s) to ad-hoc architecture hacking (2010s) to ad-hoc pre-training hacking (2020s).
20
66
753
If you want to go into CS academia, first be a lousy jazz musician for 10 years. You will never find the pay low (even in PhD), the amount to learn onerous (even in mature subfields), the feedback cruel (even from R2), or the the rejection rate high (even at NeurIPS/ICML/COLT).
16
39
714
***In the first thrilling installment, the Superheroes of Deep Learning find themselves up against an ordinary problem. Will their extraordinary abilities carry the day?***
New on Approximately Correct!
@AndrewYNg
@drfeifei
22
140
696
Guys, I have a burning question. What's the best language for data science, Python or R?
295
76
677
For as long as I've been a computer scientist, I've been convinced that I am the worst possible mathematician one can be while still hoping to do meaningful ML research. This feeling hasn't changed at all as my abilities have fluctuated.
19
31
687
Physics is searching for a *theory of everything.*
Deep learning is searching for a *theory of anything*
25
49
680
***ML Tragic Pattern*** (1) Propose algorithm that makes no sense. (2) Ignore obvious examples that illustrate the incoherence. (3) Implement it anyway (4) Fool reviewers @ a top conference. (4) Celebrate. (5) "Discover" method has problems. (6) Write "critical" paper.
18
73
675
If I ever shared how abusive & toxic my experience of the business technologies group at Tepper was, the group would cease to exist. It still rattles me every day. I couldn’t even share my joy when I escaped from that hellhole. Too traumatic to acknowledge I ever worked there.
28
46
660
***OpenAI Trains Language Model, Mass Hysteria Ensues*** New Post on Approximately Correct digesting the code release debate and media shitstorm.
14
187
611
BREAKING NEWS: Many of you are panicking about where to find more toilet paper. Turns out every general audience AI book from the last 5 years contains 250+ sheets. 🧻🚽
17
69
611
The double descent phenomenon is described in ~1000 papers & talks over past year. It's featured in at least 1 slide per talk @ last summer's Simons workshop on Foundations of DL. Why is this
@OpenAI
post getting so much attention as if it's a new discovery? Am I missing smtg?
A surprising deep learning mystery:
Contrary to conventional wisdom, performance of unregularized CNNs, ResNets, and transformers is non-monotonic: improves, then gets worse, then improves again with increasing model size, data size, or training time.
95
666
2K
25
81
613
When I grow up, my dream is to become the i-th (3 < i < N) author of a major
@Google
paper for an intern-level contribution, then set $1B dollars on fire. Later, I plan to play Call of Duty in a villa bought w secondary sales while employees pay the cost as it all implodes.
13
35
605
10x researchers
Professors, if you ever come across this rare breed of researchers, grab them. If you have a 10x researcher as part of your lab, you increase the odds of winning a Turing award significantly.
OK, here is a tough question.
How do you spot a 10x researcher?
16
90
595
Here’s a puzzling fact about the mainstream ML/AI community: a bunch of ostensibly creative dreamers were handed job security, massive amounts of cash, and unprecedented creative license—and the only thing most ppl can think to do with this freedom: train bigger networks. 🤣
How does a university based researcher keep feeling relevant in this fast changing and compute driven field?
Some good discussion here ➝
18
77
435
25
72
585
Anyone want to guess what deep learning hotshot drives this hot rod?
16
35
563
It’s hard to understate just how strange the state of “explainable AI” is today. For nearly all “local explanation” techniques, the only people who understand them would never use them, and the only people who use them do not understand them.
17
84
555
Overjoyed to announce that I have N papers accepted to the [name of international conference on X]! [no details about the work or coauthors follow]
#conferencenameYYYY
12
35
525
There’s been a rash of deep learning & VC bozos saying things like “with enough data, correlation is causation”. This is incorrect, damaging to the discourse, & makes us (empirical ML researchers/practitioners & tech broadly) look like idiots to the science-literate world.
17
62
527
New life plan: (1) launch a fledgling AI startup; (2) somehow get publicly listed with no business plan & few routes to solvency; (3) convince every hedge fund to short the shit out of my stock; (4) befriend /u/DeepFuckingValue
12
31
515
I'm all for a "slow movement" in ML publishing (see my tweet suggesting 1-yr paper moratorium last wk). But is the right avatar for such a movement a man who ostensibly authored *92* papers last year? Let's discuss (& pls keep it constructive everyone).
37
64
521
Awesome ML blog run by
@lilianweng
has fantastic exposition, clear illustrations and covers a wide spectrum of topics in classic and modern ML:
5
102
512
Overall the ML community at present does not have this problem...
12
21
509
Why is ML "taking over" so many adjacent fields, stealing the spotlight & overshadowing (often-comparable) work? The rapid pace of innovation? The talent? The tooling? I'm inclined to think the major (unsung?) factor might just be the universality of ***open-access publishing***.
32
59
501
I think most of us find Einstein's diaries upsetting, notably for indulging in ugly stereotypes when discussing his travels in Asia. At the same time, I'm conflicted about the decisions to publish dead people's private diaries. Thoughts?
74
57
468
The precarious state of “interpretable deep learning” is that we should be far more scared upon hearing that a hospital or government deploys any such technique than upon hearing that they haven’t.
19
66
490
5
38
479
Just spent a full day shadowing radiologists reading mammography and diagnostic breast cancer scans (MR, ultrasound, etc). Huge opportunities to use ML to *help* but ***wow wow wow*** are some people underestimating the work radiologists do.
22
124
478
***What were the most interesting, scientifically insightful, and coherent papers that you read in 2019??***
Rules:
1) No restriction on methodology or aim (theoretical, experimental, & applications papers welcome).
2) Do not post your own papers.
34
91
472
In most ML textbooks (& intro-y stats books), distributions fall out the sky, w/o full derivations of their precise functional forms (e.g. why the gamma function appears in the Beta). Does anyone have a great intro prob/stats books that works everything up from first principles?
25
64
470
“If you haven't run a baseline of logistic regression, you are committing algorithmic malpractice.” –
@scorbettdavies
at
@ucsantabarbara
Responsible ML panel. YES.
6
88
471
I've seen a lot of BERT tutorials, not a single one that you could read and thereafter reproduce anything close to BERT (without consulting source code). Is there a more complete tutorial out there I'm missing?
24
50
450
If
@overleaf
ever truly dies, I will quit research, move to Greece, and spend the rest of my days baking under the sun, reading sci-fi novels, & drinking away the pain.
22
23
439
Insisting that authors include a LIME/SHAP/IG/TCAV/GRADCAM saliency map in a paper shd be a disqualifying offense for reviewers. Including any such map without a powerful disclaimer (“this means nothing”) shd be a disqualifying offense for authors
Extra true for ML in healthcare
29
54
436
People often ask if academia can keep up in research with industry given the superior resources there. Honestly, 99% of *interesting* academic work *even in DL* requires < 16 GPUs. This is maybe 1/100th the cost of an electron microscope. Resources are a bit over-emphasized
21
70
423
Hey everyone. Stoked to report that we've blown away recent NLP benchmarks with a new sentence embedding: "Efficient Recurrent Neural Inverse Embeddings" Idea's simple: iteratively embed & invert sentences gains thru meta-learning magic!
#NLP
#deeplearning
20
57
424
***Uncontroversial opinion***: Geoff Hinton & Jurgen Schmidhuber both made remarkably creative (and prescient) contributions to deep learning. The credit assignment process is broken and biased, with pockets of work overlooked, but not necessarily due to malice of any individual.
11
31
422
Dear world (CC
@businessinsider
,
@Hamilbug
): stop saying "an AI". AI's an aspirational term, not a thing you build. What Amazon actually built is a "machine learning system", or even more plainly "predictive model". Using "an AI" grabs clicks but misleads
26
96
420
Do pretraining’s big wins in NLP really involve “knowledge transfer”? Are upstream corpora even needed? *Not always!!* My students
@kundan_official
&
@saurabh_garg67
show that self-pretraining (from scratch) often rivals “foundation model” performance.
8
85
417
Advice to a young academic in 2016: get on Twitter.
Advice to a young academic in 2021: get off Twitter.
15
20
415
Absolutely delighted to join the Operations Research group (joint w. MLD) at
@teppercmu
. With ML now applied in the wild, we must address both *predictions* & *decisions*. This nexus is where I see my research headed & I couldn't hope for stronger colleagues on either side.
12
15
413
Dear
@overleaf
, if you implement a feature that allows us to search for projects by collaborator names, my PhD student
@dkaushik96
will tattoo "
@overleaf
" on his biceps and name his first child
@overleaf
.
11
10
410
Don't go into a career in research for the fast cash that may come & go w. industry interest. Go into research for the freedom to wake up every day & decide that the most important thing in (your) world is to read about topic X, then do it, and then annoy people about it.
4
72
406
Super excited to announce that “Dive Into Deep Learning” now officially supports PyTorch!
Dive into Deep Learning now supports
@PyTorch
. The first 8 chapters are ready with more on their way. Thanks to DSG IIT Roorkee
@dsg_iitr
, particularly
@gollum_here
who adapted the code into PyTorch. More at .
@mli65
@smolix
@zacharylipton
@astonzhangAZ
8
241
940
9
54
400
Quarantine has made one thing clear: The real reason for conferences is to force weeklong breaks from meetings.
5
17
402
Anyone who tells you they are not jaw-on-the-floor surprised that you can have this sort of interaction with a chatbot today is lying.
13
8
391
Me: [rambles while introducing an idea, apologizes]
@dkaushik96
: "no, keep going. this is doing wonders for my impostor syndrome". 🤣🥳😅
— The Joys of PhD Advising, 2020
6
5
386
Ali Rahimi delivered a rare critical talk at
#nips2017
, likening modern ML to alchemy. Examples: brittleness of SGD to implementation changes & mystical claims like "batch norm works by reducing internal co-variate shift". Full talk now on YouTube:
3
146
379
Can someone give me tenure already so that I can write an honest book about the 2010s AI boom?
18
14
378
I can't imagine being a woman in CS seeing this week's news. I feel shocked and devastated. Foremost for the victims. But also for all the other women who have to put their guard up higher, doubting the authenticity of the professional attention they receive. You deserve better.
3
20
383
After a decade of carefully maintained social media silence, our dear visionary
@geoffreyhinton
breaks his twitter silence… to gripe about retail banking customer service.
Does HSBC UK have any ML people? HSBC will not comply with my written instructions to transfer money within the UK. Fraud detection says it must be authorized by high value transfers. High value transfers say they cannot authorize it. 7 hours on the phone so far. Help!
68
36
564
10
17
383
Academia is not special because it's perfect or because it always lives up to ideal. It's easy to point to BS work and bad processes to trash the institution. But the 5% of work that is both magical & unlikely to happen anywhere else makes the entire enterprise worth it.
7
33
379
eight yrs ago i was a jr phd student over the moon to be sharing beers with the inventor of the lstm (
@HochreiterSepp
) at neurips in montreal w my first deep learning collaborator
@davekale
. hope you get to share a human moment w someone you admire this week.
3
24
302
For all the criticism that I direct towards ML, I'm grateful to belong to a community open-minded enough to laugh at itself and flexible enough to change. Having experienced more conservative, closed-minded corners of academia, I know it's a luxury not to be taken for granted.
7
10
369
Importance weighting is widely used but ***may have no effect on deep nets*** modulo choices regarding early stopping and weight decay. This "science-y" paper with my student Jonathon Byrd identifying this phenomenon was just accepted at
#ICML2019
(1/2)
5
52
368
Interviewing Elon Musk and Mark Zuckerberg for expert opinions on AI is like interviewing the president of Switzerland abt particle physics
19
112
359
Notable: while LLMs are the singular force shaping the discipline & dominating the discourse in NLP, I saw absolutely nobody from the major LLM shops (
@OpenAI
,
@AnthropicAI
,
@MosaicML
) make an appearance at
#acl2023
.
19
27
360
Is the best Variational Autoencoder (VAE).... not actually a VAE at all?!? This paper is wild. Extremely simple idea: regularized deterministic AE + ex-post density estimation.
Results somehow both unsurprising—intuitive!—& shocking—how did people not know this for years?!?
@skoularidou
@_sam_sinha_
@zacharylipton
but that uses adversarial learning in one variant (and MMD in another).
Let me shamelessly plug RAEs
The catch is that ex-post density estimation on the latent space gets you better aggregate posterior estimation and hence sample quality
4
14
124
4
69
355
The perennial Q of "how to ensure the US continues to lead in AI?" is so easy to answer. Fully fund all PhDs in strong programs outright, grant visas to best students in the world, & let them stay afterwards. It would be the cheapest grand initiative in the history of government.
4
56
351
The reading list for pilot run of 10721: Philosophical Foundations of Machine Intelligence has been finalized and presenter notes from groups to date are up on the course GitHub for anyone interested in reading along:
13
70
355
Note to all ML authors: w k-fold cross validation, you *must* have an independent test set. Especially in healthcare, I've seen many papers this year with no test set. This qualifies as a desk reject.
17
85
348
One interesting (but incomplete) measure of the state of the deep learning bubble. While PyTorch has caught up with TensorFlow, aggregate interest in DL frameworks seems to be maybe half what it was at the peak. I can imagine many interpretations, share yours.
48
44
338
Idea of the week: ***One year moratorium on papers.*** For the entire community. A whole year of thinking without sprinting/hustling/spamming towards deadlines.
21
20
346
In latest ad campaign,
@IBM
says solution to homelessness is [wait for it...] AI. The virtuosos have hit a new peak for shameless cynical bullshit.
22
32
334
Biden is very likely going to win, but going forward, can we stop acting like
@NateSilver538
knows more than any other semi-smart blogger with lumpy prose and an undergraduate's knowledge of statistics?
31
17
336
Excited to announce some personal news: After 3 yrs as science advisor to
@AbridgeHQ
, I’m jumping in as Chief Scientific Officer. Our team of NLP/ML researchers, designers, & engineers are tackling some of the biggest pain points facing doctors & patients. More to come…
25
10
341
Before the media blitz & retweet party get out of control, this idea exists, has been published, has a name, and a clearer justification. It is called ***Counterfactually-Augmented Data*** and here's the published paper (spotlight at
#ICLR2020
).
7
60
336
@weatherbryan
@jonkeegan
Are those airplanes all in the air simultaneously? That's terrifying.
8
8
303
Help me Twitter. Best (reasonably technical) math/science podcasts?
34
43
331
Prompts are not Lipschitz. There are no “small” changes to prompts. Seemingly minor tweaks can yield shocking jolts in model behavior. Any change in a prompt-based method requires a complete rerun of evaluation, both automatic and human. For now, this is the way.
14
40
330
This fall I'm leading "CMU 10732: Robustness & Adaptation in Shifting Environments" focusing on causally structured shift, adversarial attacks, strategic classification, online adaptation, & feedback loops in recsys. Follow our readings & lecture notes:
6
58
330
"It's me Siraj, let's talk about quantum computing." =>
"It's **Maine** Siraj, let's talk about quantum **cipher**"
31
33
327
There's a handful of ML ideas that just *feel right*---perhaps due to evoking some aspect of human learning?---that keep recurring but never seem to have panned out convincingly. Here's two: (1) curriculum learning; (2) hierarchical reinforcement learning. (Dis)agree? Got others?
33
26
322
BTW, deep learning libraries are full of this—e.g., undeclared weight decay. All kinds of things lurk that seem like good ideas using the "works better in general" argument but actually create a sloppy environment where scientists don't really know what model they are using.
By default, logistic regression in scikit-learn runs w L2 regularization on and defaulting to magic number C=1.0. How many millions of ML/stats/data-mining papers have been written by authors who didn't report (& honestly didn't think they were) using regularization?
50
215
1K
13
45
321
Across all political & ideological commitments, there's a movement today away from civil discourse & towards signalling, dunking, & mindless pile-ons. It's aided (or caused) by the amplification dynamics encoded in our platforms, and it's ruining us all. We can & must do better.
15
27
311
Why have a "standard route" at all? It seems that the greatest poxes facing the ML community are intellectual homogeneity & group-think. Having everyone take too-similar paths to PhD appears (to me) the very pathology implicated. Granted, as a strange-path freak, I'm biased.
16
28
319
Amazing how many in the deep learning community can simultaneously decry the closed-minded monoculture that spurned NN research in the 90's and celebrate an even narrower monoculture in the 2020's.
3
35
310
Before you write the word "novel" 15 times in a paper, think: are you planning to work in ML for more longer than this hype cycle? Do you anticipate respectable (read *scientifically literate*) ppl reading your paper or is your audience only the VC you're presently defrauding?
15
49
307
If twitter dies, where are 70% of the world's greatest scholars going to spend 5 hours per day posting pallid platitudes?
17
15
309
Steps to get hired as an "AI researcher" in silicon valley:
1. Download a deep learning library
16
52
303
Excited to join CMU & also wistful abt leaving UCSD. They took me in for PhD 4yrs ago when I was jazz musician w/o research experience, a life-changing opportunity that I fear might become rarer as field grows more competitive & PhD programs are swamped w experienced applicants.
8
8
306
Well, there’s one thing I can never take away from
@ilyasut
. In research and in life, this man’s capacity for novelty should never be underestimated.
4
7
301
Is there any known case of anyone accessing “harmful capabilities” of an LLM that didn’t consist of knowledge already freely available and clearly described in documents on the open web?
Is the fear that we are basically just getting what we would already have if Google / Bing…
37
43
299
I live in Pennsylvania & so can you. PhD & MS applications to
@mldcmu
&
@teppercmu
open now! *Move to Pittsburgh, do great research, have a voice.*
If you work on fairness, healthcare, distribution shift, feedback loops, or NLP (but more than BERT) consider applying to
@acmi_lab
9
31
298