Aditya Kusupati
@adityakusupati
Followers
2,825
Following
1,523
Media
95
Statuses
1,478
🔬PhD.. @uwcse : @RAIVNLab ; Been places..... Done things....
Seattle
Joined March 2012
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Rio Grande do Sul
• 271028 Tweets
Madonna
• 179326 Tweets
Ole Miss
• 99839 Tweets
DAVI NO PODPAH
• 99268 Tweets
Sony
• 99092 Tweets
Porto Alegre
• 79629 Tweets
Star Wars
• 74103 Tweets
#SmackDown
• 67575 Tweets
Hope Hicks
• 65127 Tweets
みどりの日
• 47120 Tweets
スパコミ
• 43243 Tweets
WNBA
• 36846 Tweets
Mark Hamill
• 36427 Tweets
#TheAmazingDigitalCircus
• 33719 Tweets
Keyshia
• 25521 Tweets
Thiago Silva
• 18939 Tweets
Quintana
• 17921 Tweets
#Maythe4thBeWithYou
• 16033 Tweets
設営完了
• 15619 Tweets
Pomni
• 13485 Tweets
Caitlin Clark
• 12632 Tweets
Benedetti
• 11766 Tweets
Luke Skywalker
• 10425 Tweets
G7最下位の70位
• 10316 Tweets
Announcing MatFormer - a nested🪆(Matryoshka) Transformer that offers elasticity across deployment constraints.
MatFormer is an architecture that lets us use 100s of accurate smaller models that we never actually trained for!
1/9
7
110
618
Introducting🪆Matryoshka Representations for Adaptive Deployment🪆
TL;DR: up to 14× lower real-world classification & retrival costs at web-scale at no loss in accuracy & w/o any overhead across setups.
Paper:
Code:
[1/11]
6
101
493
Introducing💃AdANNS: A Framework for Adaptive Semantic Search🕺
TL;DR: Up to 90× faster nearest neighbor retrieval and 2× lower memory cost for web-scale search.
Applies to vector search at scale & improves all "retrieval" augmented models!
[1/8]
5
97
487
🤯WOW🪆Matryoshka Representation Learning enables "native support for shortening embs" &"very flexible usage"
Jokes aside, excited that
@OpenAI
serves MRL by default in v3 embedding API for retrieval & RAG!
Other models & services should catch-up soon😄
6
48
371
To be clear, we are very happy that
@OpenAI
adopted it & now even more people will continue to innovate on it. Products should use academic research!
However, from the pov of a grad student, it would have meant so much to us if there was attribution.
Thanks for all the love 🩷
Pace of progress in AI is lightning!
@OpenAI
released MRL style text embeddings, mirroring our NeurIPS '22 paper (w/ awesome folks from UW and Harvard).
However, as an advocate of open science, I am a bit disappointed with rebranding to "shortening embs" without ref to MRL 1/n
11
101
660
7
15
263
PSA(2) - prospective PhD applicants in CS.
Right time to share the "One-stop" destination for all grad application related stuff; maintained by the amazing
@kalpeshk2011
& will cover 99% of your bases/Qs.
Overwhelming? I will highlight the unmissables👇
4
49
192
Wohooooooooooooooooo!
Thank you all!!!!
@jainprateek_
@OpenAI
Hey Prateek! We did train this based on MRL - I was responsible for the blog post and it’s my bad not thinking / remembering to cite. We’re updating the blog post to add a citation now!
11
19
268
7
4
189
📢📢At the last minute, I decided to go on the job market this year!!!
Grateful for RTs & promotion at your univ.😇
CV & Statements:
Will be at
#NeurIPS2023
! presenting AdANNS, Priming, Objaverse & MADLAD. DM if you are around, would love to catch up👋
2
49
183
Multi-class classification of *L* classes traditionally need *L* classifiers.
1st thought in 2016: Can we do this w/ *log L* bits?
I finally acted on it, we propose LLC that solves 1000 class ImageNet classification with **20-bits**.
A thread🧵(1/8)
3
20
153
Updated blog from OpenAI:
Glad this was resolved, last couple of days have been pretty interesting! Thank you everyone for the support.
Back to making my job talk, so I can actually create more things like these when I get kicked out of PhD soon😜
0
7
149
Now that OpenAI's hierarchy is resolved, time to see if "Hierarchical" representations capture any hierarchy!
@ethnlshn
investigated this to find that "hierarchical" embeddings don't capture hierarchy any better than regular embeddings 🤯
Paper:
1/2
5
15
142
This is just amazing!!! Check out this seminar
@uwcse
about thriving in grad school. Great resources and a step in the right direction 🥳
Great initiative by Anna Karlin, Mike Ernst and
@_pratyush_patel
1
34
143
🤯I still cannot believe that a runaway para (pg 9) in our old paper (LLC, NeurIPS'21) lead to all the Matryoshka works🪆
At that time, I did it as I didn't have compute to train 3 models 🤣
Last couple of weeks have been surreal, thanks everyone!
LLC:
4
8
123
📢🪆MatViT-B/16 & L/16 model checkpoints & code are public - drop-in replacements that enable elastic compute for free!🔥
Try them out; let us know😉
Shout out to
@kfrancischen
for the release;
@anuragarnab
&
@m__dehghani
for the amazing Scenic library.
Announcing MatFormer - a nested🪆(Matryoshka) Transformer that offers elasticity across deployment constraints.
MatFormer is an architecture that lets us use 100s of accurate smaller models that we never actually trained for!
1/9
7
110
618
0
23
109
I am visiting
@berkeley_ai
&
@GoogleAI
for the summer working on fun ideas!!
I often frequent SF and south bay while staying in Berkeley.Happy to meet people who might be around. 🌉
DM or email me to grab coffee/gelato or talk research 🔥
0
0
108
Can we learn pruning thresholds and, in-turn, layer-wise sparsity distribution?
Introducing STR, Soft Threshold Reparameterization, a simple trick on weight tensors to achieve SOTA accuracy in sparse DNNs while reducing inference FLOPs significantly!
1/n
4
23
100
Inspired by the other similar initiatives, the students
@uwcse
started Pre-Application Review Service (PARS) to provide feedback on the CS PhD application materials.
Underrepresented students are strongly encouraged to submit.
Deadline: 8th Nov 2020.
1
49
100
I love reviewing and I am glad that the conferences continue to recognize my efforts.
My 3 conferences till now.
#NeurIPS2020
- Top 10%
#ICML2020
- Top 33%
#NeurIPS2019
- Top 400
The best tier in each of them and it just gives enough energy to do it again diligently.
1
1
75
A very nice paper on how to train text embeddings at scale:
Congrats!
@nomic_ai
@zach_nussbaum
@jxmnop
@andriy_mulyar
0
9
73
@thegautamkamath
Didn't know having high GPA in undergrad mattered in PhD apps more than being at the "best" school. Had 1 constricted admit in '17. GPA haunted even in '19; Stanford called to ask why was it so low?
IT WAS NOT! Compete with AIR top 50 to know what 8.63 is at IITB on a bell curve
3
5
67
📢🔥Next gen. Google text embeddings, Gecko🦎, leveraging knowledge distilled from LLMs & of course with🪆Matryoshka between 256-d and 768-d🪆! Very strong MTEB results 🥳
Google Cloud API 👉… Paper:
Colab:
Introducing Gecko 🦎, a new text embedding model from Google DeepMind! Distilled from LLMs, Gecko offers powerful embeddings for various NLP tasks. Gecko is now available in Google Cloud API 👉
Paper:
Colab:
10
80
353
0
7
68
Gecko elastic embeddings powered by Matryoshka🪆 announced today at
@Google
CloudNext today!
Available through Cloud (Vertex) and Google AI Studio.
0
9
62
Nice to see the application of🪆Matryoshka representations as part of
@PinterestEng
's LinkSage model used for AI systems across
@Pinterest
!!!
0
10
58
I did a thing (MRL & AdANNS -, TBD); &
@GoogleAI
thought it was useful & interesting to be in the blog articles on Google Research, 2022 & beyond
Read the last para before conclusion.
@jainprateek_
,
@RAIVNLab
,
@ShamKakade6
,
@wregss
,
@uwcse
,
@BhattGantavya
As
#DeepLearning
models become more widely used, it is increasingly important that they be both robust and efficient. Today we summarize some of our many efforts to improve
#ML
efficiency through algorithms research. →
28
119
656
2
5
58
.
@GoogleAI
India's pre-doc program is a must for anyone wanting to figure out what excites them.
If it were not for MSR India's RF program, I probably would never have found my research interests.
Of course
@jainprateek_
is amazing, and maybe you get to work with me as well 🤪
Applications now open for pre-doctoral positions at
@GoogleAI
India!
#predoc
Come spend 2 years with our ML and Optimization team working on cutting edge problems in DL understanding, efficient ML, RL, privacy!
@ManishGuptaMG1
,
@divy93t
,
@PNetrapalli
2
32
125
3
1
58
Great thread!!
Would have been amazing if it were out 3 months ago, but still an amazing thread for people eyeing a PhD in the long run.
Apart from a master's, I would also recommend RAships, long research internships and residency programs to figure out research inclinations!
I've been serving on grad admissions committees at MIT for 5 years - in EECS and Media Lab
If you want to get into a PhD at a place like MIT, here's a thread with some advice based on my observations
1/13
34
552
2K
0
4
54
Been a safe 1.8 yrs and finally the virus found me even after the booster. Mild symptoms!
4
1
51
🪆embeddings play very well with quantization!
MRL and quantization are complementary and user need not just pick one (a common misconception)!
See more in
cc:
@wregss
Would you trade 1% of accuracy to only have to store 1% of the data?
With Matryoshka Embeddings, we can drastically reduce vector dimensionality while maintaining strong accuracy. Check out this example that combines Matryoshka Embeddings with Faiss 4-bit scalar quantization 🚀
3
7
36
3
7
48
Gecko embeddings on MTEB leaderboard🔥🔥!
Best in-class for 768-d & 256-d (thanks to🪆) with a single 1B model!
Read more on how it is done in the paper:
And try it out on Colab:
📢🔥Next gen. Google text embeddings, Gecko🦎, leveraging knowledge distilled from LLMs & of course with🪆Matryoshka between 256-d and 768-d🪆! Very strong MTEB results 🥳
Google Cloud API 👉… Paper:
Colab:
0
7
68
0
5
48
Our latest work!! I will share a tweet thread soon🪆
Work co-led w/
@BhattGantavya
@wregss
Collab between
@uwcse
/
@RAIVNLab
&
@GoogleAI
Co-authors: Matt Wallingford,
@adityaasinha
,
@RamanujanVivek
, William, Kaifeng,
@ShamKakade6
,
@jainprateek_
and Ali Farhadi.
Matryoshka Representations for Adaptive Deployment
abs:
flexibility within the learned Matryoshka Representations offer: (a) up to 14× smaller embedding size for ImageNet-1K classification at the same level of accuracy
1
15
69
0
6
46
📢📢We are releasing MatFormer based OLMo (
@allen_ai
) checkpoints (up to 1.3B params & 160B tokens) and code to further open research powered by
@KempnerInst
cluster🔥🔥
Checkout our new article summarizing MatFormer as part of the Deeper Learning blog!
In our latest Deeper Learning blog post, the authors introduce an algorithmic method to elastically deploy large models, the
#MatFormer
. Read more:
#KempnerInstitute
@adityakusupati
@snehaark
@Devvrit_Khatri
@Tim_Dettmers
0
12
21
1
12
43
Thanks for the kind words Andrew! Glad people are appreciating utility of MRL.. Been a hard battle convincing people in retrieval world to use it and improve things beyond current scale!
(1) Matryoshka embeddings (): TLDR in training given an embedding of full dimensionality M (e.g. 2048), learn N different distance functions for each prefix of the embedding (e.g. l2_norm(embedding[:32]), l2_norm(embedding[:64]), l2_norm(embedding[:128]),…
2
19
172
2
1
41
The code for AdANNS💃 is out at (primarily led by
@wregss
) and has integrations with Faiss NN search library along side native GPU/CPU implementations
Check it out and let us know!
Introducing💃AdANNS: A Framework for Adaptive Semantic Search🕺
TL;DR: Up to 90× faster nearest neighbor retrieval and 2× lower memory cost for web-scale search.
Applies to vector search at scale & improves all "retrieval" augmented models!
[1/8]
5
97
487
1
10
38
A beautiful walk through how to enable first pass adaptive retrieval with Matryoshka Representations 🪆
Of course, you can further improve QPS if you can make ANNS take all the granularities, so check out AdANNS that makes it even faster! cc.
@wregss
Matryoshka embeddings allow you to "shorten" their dimensions (eg. OpenAI's v3 embeddings). But if you're like me, you probably want to know:
‣ how does shortening actually work?
‣ how are these models trained differently?
‣ can we take advantage of this in vector search?
4
18
56
2
3
36
Congrats to
@uday_kusupati
(his 1st paper in Ph.D.) and his co-authors
@GCM_EPFL
for winning an inaugural "Best Paper Honorable Mention"
@siggraph
for their work on "Umbrella Meshes: Elastic Mechanisms for Freeform Shape Deployment" 🥳⛱️☂️🌂
2
1
35
My reviewing load started to get to me and I decided to reach out to both
@CVPR
and
@iclr_conf
(not submitting) asking for a reduced load. Both of them graciously accommodated my request (Within 5 mins of emailing). Thanks!
Always ask when you might not be able to give your 100%
0
1
34
Happening now!! Hall J
#640
MRL!!! 🪆🪆🪆🪆🪆 FTW
Excited to be back for an in-person
@NeurIPSConf
conference after Feb'20!!
I will be around the whole week. Feel free to DM me for catch up or a chat. Drop by MRL's 🪆 poster (Tue 11a Hall J
#640
) w/
@wregss
,
@adityaasinha
&
@ShamKakade6
.
Also at
@GoogleAI
booth on Tuesday!
1
6
23
0
1
34
Had fun chatting with
@CShorten30
@ZainHasan6
@zach_nussbaum
on everything 🪆🪆
Do checkout the podcast for the tidbits!
Matryoshka embeddings are one of the most recent and powerful innovations in Vector Search! 🪆
The core idea is to learn multiple vectors of increasing lengths, which opens up all sorts of new opportunities for the mechanics of nearest neighbor search!
I am SUPER excited to…
7
42
160
2
4
32
Dear ML twitter,
@wregss
, a master's student at UW ECE had his
@NeurIPSConf
grant application denied.
Does anyone have a complimentary registration that could be transferred to him? He will be applying for PhD this cycle & is terrific at getting things to work!! Thanks.
1
3
31
After a hectic quarter ending w/ a robotic
@icmlconf
video recording. Yeah, STR got in. Excited to explore new things
@NVIDIAAI
w/
@FidlerSanja
& Antonio Torralba.
A great summer in Toronto, Austria & India? - Looking forward to recreating it from Seattle
1
1
30
PSA - prospective PhD applicants in CS
Looking for a clean CV template? I have open-sourced mine and people seem to like it😅
Template:
Built this over a span of 2 years, by editing and adding to an amazing .cls file. Has most blocks we typically need.
0
4
29
MSR RF program is the reason I am doing an ML PhD today. It was 2 beautiful years filled with amazing learning experiences and great mentors!!
I strongly recommend RF program to anyone wanting to venture into research.
Thanks to
@jainprateek_
@SriramRajamani
among many others!!
Microsoft Research India invites applications for its 2-year Research Fellow program. Recent graduates and Bachelors/Masters students majoring in CS or a related area may apply. Deadline Jan 15, 2021. Please RT.
6
74
219
0
2
29
GRE score sometimes was the reason for not making a PhD offer😕 It doesn't factor PPP & there is nothing worth INR 14K (Indian PhD student stipend: 30K for comparison) in those stupid under-equipped noisy halls.
Persnickety is a GRE word which describes it so well
#AbolishGRE
2
3
27
Go use🪆in Sentence Transformers!
🔥Sentence Transformers v2.4.0 is released! It introduces Matryoshka Embedding models (training & inference), 2 new state-of-the-art loss functions, prompt templates, instructor model support & more. See the🧵
6
93
377
1
1
26
This is the truth, the whole truth, and nothing but the truth. This whole thread is the reason that show can't be looked at like a joke. The trailer was enough to give me jitters.
Things are still the same for most of the girls in my hometown (a tier 3 town in India).
2
1
24
Adding to the meltdown. Just got a desk reject with default feedback (as
@sineadwilliamso
mentioned) from NeurIPS for a paper we are super proud of.
I reviewed 6 papers last week & was fully convinced that desk rejects didn't go out looking at a couple of papers in my pool 🤷♂️!
1
0
25
We will be presenting STR at
#ICML2020
this Thursday at noon and again at midnight PDT.
Paper:
Code:
Video:
Please reach out here or in the rocket chat to discuss anything!!🥳🥳🥳
2
2
25
You're meant to say "Clippy is back".
As one of the most unsurprising moves in tech,
@Microsoft
has announced that they will incorporate all of
@OpenAI
tools into their products. In particular, this means a wide availability of ChatGPT in various Office products.
1/9
45
205
2K
1
0
23
Starting the day with a
#NeurIPS
rant!!
A beautiful pair of uninformative yet highly confident and low scoring reviews 🫠
0
1
22
Excited to be back for an in-person
@NeurIPSConf
conference after Feb'20!!
I will be around the whole week. Feel free to DM me for catch up or a chat. Drop by MRL's 🪆 poster (Tue 11a Hall J
#640
) w/
@wregss
,
@adityaasinha
&
@ShamKakade6
.
Also at
@GoogleAI
booth on Tuesday!
Introducting🪆Matryoshka Representations for Adaptive Deployment🪆
TL;DR: up to 14× lower real-world classification & retrival costs at web-scale at no loss in accuracy & w/o any overhead across setups.
Paper:
Code:
[1/11]
6
101
493
1
6
23
Thanks
@ducha_aiki
for sharing our new work led by Matt Wallingford!
We will share a tweet thread with webpage and code soon!!
TL;DR: Transferable/shared object-centric representations across scenes using NeRF -- generalizes well from synthetic to real-world datasets!
Neural Radiance Field Codebooks
Matthew Wallingford, Aditya Kusupati, Alex Fang, Vivek Ramanujan, Aniruddha Kembhavi, Roozbeh Mottaghi, Ali Farhadi
tl;dr: discovered 3D assets for NERFs?
1
15
69
1
0
22
I am honored to be a student (still am) of this superstar!!
Congrats
@jainprateek_
🥳🥳
#YAA
Dr. Prateek Jain (
@jainprateek_
) received his bachelor's degree in Computer Science & Engineering from
@IITK
in 2004, and subsequently went on to earn his doctorate degree from the University of Texas in Computer Science & Engineering in 2009.
4
4
64
0
0
22
Funny finding these "Matryoshka Transformer" dolls. 🪆🪆
Announcing MatFormer - a nested🪆(Matryoshka) Transformer that offers elasticity across deployment constraints.
MatFormer is an architecture that lets us use 100s of accurate smaller models that we never actually trained for!
1/9
7
110
618
1
0
21
Krishna was and is a guiding light for me whenever I sought guidance in research and life!
Super excited that he is going back to Indian academia --
@iitmadras
is lucky!!
Go work with him and def talk to him at NeurIPS!
I’m thrilled to announce that I'll be joining
@iitmadras
as an Assistant Professor in April 2024!
I’m immensely grateful to my amazing mentors, family, and friends for their unwavering support. (1/4)
49
58
2K
0
0
21
Co-led w/
@wregss
(headed for a PhD
@WisconsinCS
)
Joint work w/ Sharan, Alan,
@sysnlp
,
@ShamKakade6
,
@jainprateek_
& Ali Farhadi.
@uwcse
(
@RAIVNLab
,
@uw_wail
&
@uwnlp
) &
@GoogleAI
Find out more about💃AdANNS🕺in the🧵below.
[2/8]
1
1
21
🎉🎉🎉🎉. I got reimbursed for
@icmlconf
😋, same $25.
My (incoming) student wanted to register for
#STOC2020
. I told him to go ahead and I'll pay for it with my grants. He said not to worry about it, since it's only $25 for students. But I told him this (see gif). Message: Your advisor is there to support all your academic expenses.
11
27
572
0
1
19
As always
@ak92501
beat us in posting about this to the Twitterati!! Thanks😅
Please do comment or reach out to us in case you have any suggestions or comments on the work!!
Matryoshka Representations for Adaptive Deployment
abs:
flexibility within the learned Matryoshka Representations offer: (a) up to 14× smaller embedding size for ImageNet-1K classification at the same level of accuracy
1
15
69
1
2
20
Visit days change your preferences and opinions a lot!
Have a fun and make a not so obvious but important decision!
It is currently PhD visit days at UW. Choosing among schools for a PhD is a tough choice. I wrote a blog post about some ways to think about this choice to make it easier and to find the school that is the best fit for you:
0
19
108
0
1
19
See y'all at
@NeurIPSConf
(virtually 🥲)
Multi-class classification of *L* classes traditionally need *L* classifiers.
1st thought in 2016: Can we do this w/ *log L* bits?
I finally acted on it, we propose LLC that solves 1000 class ImageNet classification with **20-bits**.
A thread🧵(1/8)
3
20
153
2
0
19
These smaller MatFormer submodels are way more consistent with the largest model than independently trained models.
This (1) enables consistent deployment across scales and (2) boosts inference optimization techniques like speculative decoding (6% speedup over baseline) 4/9
1
0
19
1/5 Happy to share our recent work: RNNPool, an RNN based pooling op/layer to replace memory-hungry Conv blocks without losing accuracy while reducing working RAM in CNNs
w/ Oindrila Saha, Harsha Simhadri, Manik Varma &
@jainprateek_
from
@MSFTResearch
2
3
18
Amazing initiative, I am a volunteer and I am looking forward to the event and learning the perspectives.
Extraordinary opportunity for Indian ML Eco system.
Delighted to announce
#MLinIndia
social at
#NeurIPS2021
.
Do attend if you are interested in exploring long term career opportunities in ML in India.
More details:
RSVP:
@iKDD_News
@sandeepjuneja66
@amt_shrma
@shouryaroy
1
51
193
0
1
18
The speed at with Nomic is shipping is insane!
More 🪆Matryoshka🪆embeddings to play with -- now all public!
Check out their visualization, there are some things I couldn't explain, maybe the community can help us with that 🔥
Announcing Nomic Embed v1.5 🪆🪆🪆
- Variable sized embeddings with matryoshka learning and an 8192 context.
- Outperforms OpenAI text-embedding-3-small across output sizes.
- Open source, open training code, open data.
Day 0 in
@LangChainAI
,
@llama_index
and
@MongoDB
11
84
477
1
3
18
So this is the way I become part of an Ivy? Congrats
@ShamKakade6
!!!
1/21 Banner year for Harvard CS!
New hires include Sham Kakade
@ShamKakade6
and Fernanda Viegas
@viegasf
(joining
@wattenberg
), as well as David Alvarez-Melis, Anurag Anshu
@AnuragAnshu4
, Sitan Chen, and Jonathan Frankle
@jefrankle
5
10
198
1
0
18
When pretraining for say 2048-d representation, MRL🪆also optimizes for a few (O(log(d))) lower dimensions (8, 16, 32..) to solve the same task w/ equal weighting.
That's it! That is all the algorithm.
Works across setups, models, tasks, modalities w/
default hparams!
[4/11]
1
2
17
@MSFTResearch
's v3 of EdgeML repo: is released. It contains scalable CUDA supported custom RNNCells along with the
#NeurIPS19
paper on ShaRNN and
#BuildSys19
paper on MSC-RNN. A combined effort of many people over 2.5 years 🤩
1
4
18
It’s simple - we jointly optimize for just n=4 transformers of different sizes nested in the same weight space.
With a trained MatFormer model, we can Mix’n’Match model capacities across layers to get 100s of accurate models. 2/9
1
0
18
@owencm
@jainprateek_
@OpenAI
Thanks for being candid with us Owen!
Glad this could be served to the users globally 😄🪆🪆🪆
1
0
17
MatFormer opens up many exciting directions in elastic deployment while being cheaper to train than all the baselines!
Co-led w/ amazing
@Devvrit_Khatri
&
@snehaark
Collab w/
@Tim_Dettmers
@kfrancischen
@inderjit_ml
Yulia
@HannaHajishirzi
@ShamKakade6
Ali &
@jainprateek_
8/9
1
0
17
Check out this awesome work by
@Tim_Dettmers
!
Tim's papers are always so grounded for real-world utility!! Very few people have the eye for identifying the actual problem and then solve it.
I am excited to share my latest work: 8-bit optimizers – a replacement for regular optimizers. Faster 🚀, 75% less memory 🪶, same performance📈, no hyperparam tuning needed 🔢. 🧵/n
Paper:
Library:
Video:
18
285
1K
0
0
17
@Mitchnw
from
@uwcse
presented their work on Discovering Neural Wirings
@NeurIPSConf
. The poster was a crowd puller. The paper provides a new way of thinking about connections in deep neural networks. Their accompanying demo was a super cool visualization!!!
2
1
17
.
@ssgrn
& I have been seeing some very interesting results through this survey at
@uwcse
. We would like to see if things hold the same at scale!!
Survey:
RTs appreciated 🥳🤓😇
Twitter-verse!
@adityakusupati
& I are doing a user study to understand how people perceive Internet text, possibly generated by machines. We'd appreciate if you could take our fun 6-question survey, which should take about 15 mins. Thanks a ton! Survey:
2
12
21
0
6
16
Co-led w/
@BhattGantavya
,
@wregss
Joint work w/ Matt Wallingford,
@adityaasinha
,
@RamanujanVivek
, William, Kaifeng,
@ShamKakade6
,
@jainprateek_
, & Ali Farhadi.
@uwcse
(
@RAIVNLab
&
@uw_wail
) &
@GoogleAI
Find out more about🪆Matryoshka Representations🪆in the🧵below.
[2/11]
1
1
16
@bneyshabur
@bneyshabur
very elegant work!!
Also, check
@tri_dao
's Kaleidoscope work, where they learn a bunch of internal structures we fix these days!!
1
1
15
Check this accessible blog post by Aniket all about 🪆🪆
With all the recent hype around 🪆Matryoshka Representation Learning 🪆(Thanks
@OpenAI
!), I finally put my longstanding plan of writing a detailed blog about MRL to action
This blog is NOT a paper walkthrough (see
@RitvikRastogi19
for that!)
(1/7)
6
24
168
0
1
16
This paper changed some of my thoughts on how to think about LLMs!
Highly recommend it to everyone 🔥🔥
New work by us on Large Language Models - how and why they work, and what is “in-context-learning”
We show that (to quote Dave Blei when he saw our work) “ICL is not magic: it is (consistent with) LM smoothing and Bayesian statistics!” (1/n)
3
21
123
2
0
16
AI residencies are newer versions of the RF program!! Amazing program and great people :D
How can young students prepare for a career in research? Join me as I speak to
@shrutirij
and Vivek Seshadri about MSR India’s Research Fellow program.
1
15
58
0
1
15
The good old Adaptive Retrieval (two-phase) using 🪆 implemented in
@LangChainAI
Use it for more saving and better performance!
🪆Matryoshka Retriever
A recent blog post by
@supabase
described a new technique for higher performance retrieval, without compromising accuracy. So we've just released a new retriever in LangChain.js 🦜🔗 which implements this exactly!
Use any vector store, and two embedding…
6
102
462
0
1
15
We find that MatFormer scales as reliably as vanilla Transformers and that we're able to fit a single scaling law for all MatFormer submodels. 5/9
1
0
15
In conclusion, AdANNS achieves SOTA accuracy-compute trade-off for the two main ANNS building blocks: search data structures (AdANNS-IVF) and quantization (AdANNS-OPQ).
We shall release the code shortly at
Paper:
1
0
15
For a 2.6B decoder-only MatFormer Language Model (MatLM), the optimized smaller models are as accurate as baselines with even more "free" models that improve predictably with scale! 3/9
1
0
15
@aravindr93
.
@aravindr93
how did this go so viral? 😋😅 But isn't it the same feeling every year though?
So much imposter syndrome on looking at these apps!!
1
0
15
MatFormer can be further extended to Vision Transformer-based encoders (MatViT), where we also see that we can use Mix’n’Match to extract models that span the accuracy-vs-compute curve. 6/9
1
0
15
He was a great friend and I am still unable to process what happened. Please consider contributing to the fund?!
We will miss his infectious smile, and the warmth and energy he brought to our research family. We are raising funds in personal capacity to support his family and to contribute to a future memorial objective, here:
3
26
197
0
1
15
UW needs serious disambiguation
University of:
1) Washington -
@UW
2) Wisconsin -
@UWMadison
3) Waterloo -
@UWaterloo
4) Wyoming -
@UWyonews
5) Windsor -
@UWindsor
6) Warwick -
@warwickuni
7) Worcester -
@worcester_uni
Too much confusion. Add if I missed any!!
1
1
14
TIL, Mercury is the mostest closest planet to Earth😱. Also, Mercury is the mostest closest planet to every significant object in the solar system. 🤯
Mostest closest == on average.
Apparently, this wasn't published until 2019!!
2
0
13
Finally, MatFormer enables truly elastic adaptive retrieval for the first time. MatFormer allows for elastic query-side encoders while having a fixed database index for large-scale retrieval. Why? MatFormer preserves the metric space, unlike independently trained models. 7/9
1
0
14
Acknowledge the safety net, aim high, be brave and just break the limits!!
Congrats
@jainprateek_
!!!
#YAA2021
#iitk
Dr. Prateek Jain (
@jainprateek_
) was awarded the prestigious IITK Young Alumnus Award 2021 by
@IITKanpur
on its Foundation Day, 02 Nov. 2021.
1
4
35
0
0
14
Across
@UTAustin
@RAIVNLab
@uwnlp
@GoogleDeepMind
@GoogleAI
@hseas
w/ support from
@orf_bnw
@tduerig
@_arohan_
@LukeZettlemoyer
@ManishGuptaMG1
Rahul Sukthankar &
@JeffDean
There is so much more that can be done by enabling elastic compute for generation and search!
9/9
1
0
14
Something very natural and what quite a few students have done for their deep learning course projects in past few years!
Good to see this used in real-world and especially in Seattle!
Post-event traffic congestion around stadiums is a problem worldwide. Today, in partnership with the Seattle Dept. of Transportation, we demonstrate a potential simulation-based plan calibrated for the specific dynamics of the area for better traffic flow.
14
61
279
0
0
14
I was meaning to say the "emeritus" thing somewhere in the tweetstorm, glad
@uwcse
made it public.
I think the discussion went off rails really fast and went into rubbing on the old but healing wounds. The language was insensitive and the logic was non-existent!!
#UWAllen
leadership is aware of recent “discussions” involving Pedro Domingos, a professor emeritus (retired) in our school. We do not condone a member of our community engaging in a Twitter flame war belittling individuals and downplaying valid concerns over ethics in AI. 1/11
30
217
2K
0
0
13
Approximate nearest neighbor search (ANNS) powers web-scale vector-based retrieval (
@pinecone
,
@weaviate_io
,
@vespaengine
).
However, ANNS pipelines use a fixed high-dimensional "rigid" representation for all components (768d embedding)➡️computationally expensive retrieval.
[3/8]
1
0
13
Check out our recent dataset release -- 2.8T covering 419 languages led by
@snehaark
!! It was fun understanding the sources of these datasets and auditing them.
Excited to announce MADLAD-400 - a 2.8T token web-domain dataset that covers 419 languages(!).
Arxiv:
Github: 1/n
24
138
809
0
1
13
Woww!! Our
#BuildSys19
paper on low-resource RNNs for radar classification won the "Best Paper Runner-Up" Award. Real-time on-device deep learning on ARM Cortex-M3. 🤩
Excited to share our work on "Multi-Scale Cascaded RNNs for Radar Classification" (oral & demo at BuildSys19) .
Collab
@MSFTResearch
+
@OhioState
+
@iitdelhi
to enable DL based noise rejection and target classification on resource-constrained radars.
0
0
1
1
0
13
Really cool and blazing fast!! Great work
@jefrankle
and
@DbrxMosaicAI
team!
P.S: With all the meme game on folks knowing about best open weights model in advance -- I kinda knew 😅😜
Meet DBRX, a new sota open llm from
@databricks
. It's a 132B MoE with 36B active params trained from scratch on 12T tokens. It sets a new bar on all the standard benchmarks, and - as an MoE - inference is blazingly fast. Simply put, it's the model your data has been waiting for.
34
267
1K
1
0
13
Will be attending
@NeurIPSConf
this year as well, not as an author but as a "Best Reviewer" 😅😝. Hit me up if you want your next big idea be reviewed early. Let's grab some coffee and chat about anything and everything ✌️✌️☃️. Be ready for an inquisitive visitor at your posters.
1
0
13