Jimmy Lin
@lintool
Followers
14K
Following
0
Media
341
Statuses
4K
I profess CS-ly at @UWaterloo about NLP/IR/LLM-ish things. I science at @Primal. Previously, I monkeyed code for @Twitter and slides for @Cloudera.
Joined February 2010
Reviewers automatically assume that simple is not novel. This is sheer laziness. Yes, it may be simple and obvious in retrospect, but someone had to have that insight first. Simple is good. Simple is robust, easy to implement and reproduce, broadly applicable, etc.
52
485
4K
DAAM. You saw it here first! Attribution maps for Stable Diffusion based on upscaling and aggregating cross-attention activations in the latent denoising subnetwork: example for "an angry, bald man doing research" below - demo at
14
147
868
GPT-4 and its ilk are awesome for rapid prototyping and one-offs, but at the end of the day, enterprises will deploy far smaller distilled models in production. Here's my contrarian take -.
28
150
719
So, CV researchers are looking at transformers and NLP researchers are looking at CNNs (again). What a strange world.
16
48
703
Still cropping and modifying BERT diagrams from Devlin et al. (2019)? I spent several hours redrawing BERT in PowerPoint so you don't have to. Perfect for use in presentations, papers, etc.! Happy to share under Releasing under CC BY 4.0
15
131
693
Following the AI Residency program by Google, Facebook, Microsoft, Uber, etc., I'd like to start the Waterloo AI Residency program. It's called grad school.
5
45
336
"NLP makes IR interesting and IR makes NLP useful!" - slides from my #sigir2020 summer school talk at: Get your rotten tomatoes and eggs out!.
9
49
296
Happy to share an early draft of "Pretrained Transformers for Text Ranking: BERT and Beyond", our forthcoming book (tentatively, early 2021) by @lintool @rodrigfnogueira @andrewyates
3
59
265
My (contrarian?) take: prompt engineering is programming in natural language. We've tried this before, with attempts dating back decades. Recent advances do not change the fact that natural languages are ambiguous, imprecise, under-specified, highly contextual, etc.
10
20
192
RAG is all the RAGe these days, but we (still) don't quite know how to evaluate it properly. This year, we are taking a stab at it in the context of TREC, building on 30+ years of experience in evaluating IR systems.
2
46
188
Case in point: "Passage Re-ranking with BERT" by @rodrigfnogueira and @kchonyc was never accepted anywhere because of the "too simple, not novel" laziness. Yet that paper is LITERALLY cited in every single BERT-for-ranking paper ever published since.
1
8
170
Presenting. RankVicuna 🦙 - the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting! Brought to you by @rpradeep42 and @sahel_sharify
5
33
171
Not novel. Not novel. Not novel. Reviews seem to be generated with a simple n-gram LM.
4
8
159
On the subject of current citation formats being, essentially, racist: I have two students (current/former) with surname Wang, 2 x Liu, 3 x Yang + 1 collaborator; ~dozen x Zhang in my collaboration circle. A citation like [Zhang et al. 2019] is unhelpful.
18
27
149
I've thought long and hard in recent years about building a culture of reproducibility in research. Over the holidays, I had a chance to organize my thoughts in a piece that has been on the back burner for many months. I'd welcome your comments!.
7
21
154
With so much noise from ChatGPT, LLMs, and the "we're all going to die" crowd, my students have (understandably) been experiencing existential angst, asking me about the implications for NLP and IR.
10
30
151
tl;dr - in the enterprise context, you'll start with rapid prototyping using GPT-4 (or another LLM) but eventually end up with a far smaller but just as capable specialized distilled model. That's the journey I see from PoC to prod.
8
18
143
New entrants into the camelidae family 🦙 for retrieval! @xueguang_ma presents RepLLaMA (a dense retrieval model) and RankLLaMA (a pointwise reranker) fine-tuned on (you guessed it!) LLaMA for multi-stage text retrieval:
5
18
140
Introducing RankZephyr 💨 - a fully open-source zero-shot listwise reranking LLM that achieves effectiveness parity with GPT-4! Brought to you by @rpradeep42 and @sahel_sharify!
4
25
132
"Pretrained Transformers for Text Ranking: BERT and Beyond" with @rodrigfnogueira and @andrewyates - it started on June 18, 2020 and culminates here with the official publication. Enjoy! institutional subscribers: retail orders:
1
28
129
My (contrarian?) predictions on ChatGPT, Bard, and its ilk: Regarding the two biggest problems today, (1) hallucinations and (2) toxicity, the first will be transient (i.e., solved relatively shortly) and the second will be perpetual (i.e., will never be solved). Rationale:.
6
25
127
Those who argue that curation is the answer to "more ethical AI/ML/LMs" come across as intellectually naive. Archivists have been grappling with this issue, literally, for millenia. They'd be well advised to consult some of the literature from that field.
6
19
118
I'll go on the record with perhaps another contrarian opinion: Lucene HNSW is the future and (pure) vector DB vendors are in trouble. Why?.
5
32
119
We are scheming to write a paper where the author list is Jimmy Lin, Jimmy Lin, Jimmy Lin.
5
3
111
I'll conclude by being constructive. As an AC, I have overridden this "not novel, too simple" garbage on more than one occasion. In some cases I have spent hours pouring over the literature to determine if this paper was indeed the first to have a particular insight. 1/2.
1
2
108
Dense retrieval without requiring a dedicated vector DB! Here's a guide on how you can take OpenAI ada2 embeddings (on MS MARCO passages) and perform retrieval directly using Lucene with our group's Anserini toolkit
3
17
112
What a sad state of affairs we find our field in: rejected because you didn't STOA; rejected because you STOA'ed (just leaderboard chasing, no insight); flag plant on arXiv, rejected (reviewer cites your paper as evidence of lack of novelty); don't arXiv, your idea is scooped.
3
17
105
The modern search landscape confusing you? Dense retrieval, sparse retrieval, transformer-based rerankers, multi-stage architectures, nearest neighbors, HNSW, blah blah blah. My attempt to sort it all out in a single conceptual framework:
1
22
104
Prompt-decoder LLMs for listwise reranking too large for you? Introducing our new LiT5 family of listwise reranking models: nearly as good but *much* smaller. Yup, T5's still got tricks to offer!
3
16
98
Another addition to the "X is all you need" genre of papers: We took OpenAI embeddings of MS MARCO passages and stuffed them into Lucene - turns out you don't need fancy schmancy vector stores for dense retrieval! Lucene will do.
Dense retrieval without requiring a dedicated vector DB! Here's a guide on how you can take OpenAI ada2 embeddings (on MS MARCO passages) and perform retrieval directly using Lucene with our group's Anserini toolkit
9
15
99
This includes checking with PC chairs on whether the previous work and the paper in question are by the same authors. And if I do recommend rejection based on novelty, it is with a citation to a paper confirmed not to be by the authors of the same paper under review. 2/2.
6
4
88
With @rodrigfnogueira and @andrewyates we're happy to share the revised version of our book "Pretrained Transformers for Text Ranking: BERT and Beyond" - significant updates to transformer-based reranking models and dense retrieval techniques!.
4
26
97
Presenting AfriBERTa, a pretrained LM for 11 African languages by @Kelechukwu_ What's neat is that its pretraining corpus is 0.04% that of XLMR (<1GB), but AfriBERTa performs just as well on downstream tasks! To appear at MRL workshop at #EMNLP2021.
2
18
92
Slides from my recent talk: A Conceptual Framework for a Representational Approach to Information Retrieval:
1
12
91
By knocking out two major datacenters, road construction work has literally brought down a non-trivial fraction of all deep learning experiments being conducted in Canada.
3
4
82
It's official! I'm leaving Maryland to take up the David R. Cheriton Chair in the School of Computer Science at the University of Waterloo!.
40
12
81
Yesterday @rodrigfnogueira @andrewyates and I wrapped up the final preproduction version of "Pretrained Transformers for Text Ranking: BERT and Beyond" - posted on arXiv as v3: now in the hands of @MorganClaypool and will be in print soon!.
1
16
82
#ACL2023NLP (or #ACL2023) is a great opportunity to plug what is shaping up to be perhaps the largest collection of core NLP faculty in Canada 🇨🇦🍁 at Waterloo @UWCheritonCS - joining @WenhuChen and me next year will be @fredahshi @hllo_wrld @yuntiandeng! Come find us to chat!.
1
12
84
Thanks to the tremendous effort of @edwinzhng @1729_gupta @kchonyc we're proud to present the Neural Covidex, our updated AI-powered search interface to @allen_ai 's COVID-19 corpus: Powered primarily by Lucene, T5, and BioBERT.
3
41
79
Introducing. SegaBERT! by @Richard_baihe et al. intuition is to introduce hierarchical position embeddings (paragraph, sentence, tokens) to better capture context during pretraining: simple idea, fairly large gains!.
1
20
78
Why I hate doing reimbursements: the default assumption by @UWaterloo is that you're a criminal trying to embezzle money from research accounts. Maryland was more sane. What's your experience been like at other places?.
16
0
70
If you're interested in dense retrieval, you'll want to check out this DPR replication effort led by @xueguang_ma tl;dr - BM25 is better than the original authors made it out to be, and free QA boost with better evidence fusion!.
5
6
76
BERTserini: combining the magic of BERT with Anserini for end-to-end open-domain QA:
2
22
74
@colinraffel You have to grow a grey beard first. And don't say "bag of words". Dress it up as "heuristically weighted sparse representations".
1
0
71
"Should You Take My Advice?" I recently wrote up this essay on how I communicate with my students. Maybe it applies more broadly to other advisors as well? Feedback/comments/questions welcome!
1
12
75
Wanna train a multilingual dense retrieval model but confused what to do? For example: Which backbone? Pre-fine-tune? Use non-target language data? Here's a helpful guide that begins to compile together best practices:
1
13
71
I expect this will generate some discussion. "The Neural Hype and Comparisons Against Weak Baselines"
2
25
67
We've written up a description of the Neural Covidex and shared a few thoughts about our journey so far in a submission to the ACL 2020 COVID Workshop: Comments and feedback welcome! #NLProc #acl2020nlp.
1
14
68
New work by @ralph_tang @crystina_z @xueguang_ma adds yet another prompting technique to the mix: *permutation* self-consistency prompting to overcome positional bias in LLMs. Useful for listwise ranking. read all about it!
0
10
69
I think we've reached the end of the road in how we currently perform inverted indexing. Check out our benchmark of Lucene indexing performance and analysis:
1
16
67
We're celebrating the opening of the new Data Systems Lab at UWaterloo today! First innovation: we've learned to turn off gravity.
2
12
60
PSA: I say to my students - if you're working on NNs and DL, there are at least 5 PhD students at Tsinghua, Peking, Jiao Tong, Zhejiang, . working on your idea right now. Be paranoid, execute, and publish!.
3
24
61
Our group has this server with 100TB disk. and it's always full. Why? These dense retrieval models take up so much $#!& space. But @xueguang_ma et al. came up with simple compression solutions, to appear in #emnlp2021
2
11
68
This came in the mail today. It weights 37 pounds. I told my wife, "it's for the kids". Of course, that's a lie.
4
2
61
At the outset, zero-shot effectiveness of LLMs will be impressive, but it'll likely not be "good enough". Improving the model requires you to more precisely characterize the task - that is, you need what "in the old days" we'd call a task description and annotation guidelines.
2
1
64
I've written about anti-Asian bias and model minority issues before: . and I've gotten the eye-roll reaction of "why don't you sit down since you've got it so good already". which is exactly the problem.
0
10
64
New work on using doc2query for summarization by @rodrigfnogueira et al. - works surprisingly well! Samples from CORD-19 corpus related to COVID-19 below.
0
15
64
The path will be from expensive general-purpose models (e.g., GPT-4) to cheaper specialized distilled models (e.g., encoder-only). Here's the progression I see -.
2
2
61
Another instance of the disconnect between academic reviewing and real-world impact: tl;dr - our keyword spotting model in JavaScript now powers wake word detection "Hey Firefox!" in Firefox Voice:
3
10
59
Thanks to @edwinzhng and @1729_gupta our Anserini IR toolkit can now search @allen_ai 's COVID-19 corpus. - @kchonyc 's connected it up to SciBERT and bam, we have a two-stage neural ranking pipeline! Join and build on our work!.
3
22
62
Interested in large-scale machine learning (#hadoop and otherwise)? I recommend this tutorial at #KDD2011 :: http://t.co/NlEzGEe.
0
33
60
LLMs are missing a critical ingredient. and @Primal knows what it is! (Hint: knowledge graphs and neuro-symbolic approaches) Here's a writeup of the journey so far, featuring CEO @YvanCouture - oh btw, I'm the CTO.
2
5
60
"I couldn't do my assignment because GitHub is down" will be the next "my dog ate my homework".
2
47
58
To every university president who's sent mass email condolences (offering support, solidarity, etc.) after previous racially-motivated hate crimes before. we're waiting.
6
4
56
Well, since you asked. a (short) history lesson on multi-stage ranking in IR. .
1
8
56
And finally. with a fine-tuned model: "Why do I need all those billions of parameters?" Cost pressure is a powerful economic driver. Distillation (along with pruning, quantization, etc.) provides the answer (and will likely yield safer models).
1
1
54
In preprint of our #sigir2019 short we conducted a meta-analysis of 100+ papers reporting results on Robust04. tl;dr - weak baselines are still prevalent (both neural and non-neural models).
5
17
53
Want to replicate our (@rodrigfnogueira @victoryang118 @kchonyc) doc2query work (currently) sitting on top of the MS MARCO leaderboard? Have we got code for you! paper:
0
12
53
Further evidence of the one-sidedness of recent attacks on pretrained language models: the critics have conveniently forgotten about the benefits they bring to disadvantaged populations.
11
4
53
We've released a new version (v0.11.0.0) of our Pyserini Python toolkit to support replicable IR research, now providing first-stage retrieval for sparse, dense, and hybrid representations. Our new arXiv paper provides an overview:
2
11
52
I am satisfied to end a recent paper thusly: Finally, in our collective frenzy to improve results on standard benchmarks, we may sometimes forget that the ultimate goal of science is knowledge, not owning the top entry in a leaderboard.
3
8
48
Tutorial slides to go with the book!
Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond" are available here: with @andrewyates and @lintool.
0
12
52
Apparently, I was recognized as an outstanding area chair at #emnlp2020 and didn't realize it until now. #humblebrag (do people even use this hashtag anymore?)
1
1
51
It's not every day you land a $1M (CAD) grant. announcing our Archives Unleashed 2 project led by @ianmilligan1.Looking forward to working with @ruebot @jefferson_bail @SamVFritz over the next few years!
2
1
48
Our latest study of BM25 variants, including Lucene's weird doc length encoding, with @kamphuis_c @arjenpdevries @srchvrs tl;dr - it's okay!
0
18
51
Sparse or dense representations for retrieval? Or hybrid? psssssh, says @jacklin_64 - neither! Densify sparse lexical reps and tack on dense semantic reps: best of both worlds and simplified infrastructure also (no need for HNSW or inverted indexes!)
1
7
47
The Iron Triangle of LLMs: capable models, low development costs (CapEx), low inference costs (OpEx). pick two, because you can't have all three! Examples? OpenAI API = low CapEx, high OpEx; BloombergGPT = (very) high CapEx but flexibility in OpEx depending on deployment.
2
7
52
We've connected Anserini to Solr to Blacklight to present a search frontend to @allen_ai 's COVID-19 corpus! Check out - awesome work by @edwinzhng and @1729_gupta.
4
18
47
It's hard to build usable software, but tweets like this make all the blood, sweat, and tears worthwhile. Credit goes to an awesome team!.
In the Information Retrieval course I let my students pick the IR toolkit of their choice among all the solutions we have available as research community. Clear front-runner by a mile was Pyserini: In big part thanks to its elaborate documentation!.
2
2
48
MS MARCO v2 datasets are available for TREC 2021! Anserini baselines are available here:
0
6
47
I'll be giving a talk on a Conceptual Framework for a Representational Approach to Information Retrieval on April 5, 4pm PT as a Pinterest Labs Tech Talk @PinterestEng. RSVP and learn more here!
0
5
49
Happy to share Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in 11 languages by @crystina_z @xueguang_ma @ShiPeng16 tl;dr - think of this as the open-retrieval condition of TyDi. Paper: Data:
2
12
47