Aleksa Gordić 🍿🤖
@gordic_aleksa
Followers
19,445
Following
217
Media
728
Statuses
3,779
proud father of 16 A100s & 16 H100s flirting with LLMs, tensor core maximalist x @GoogleDeepMind @Microsoft
silico
Joined September 2017
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Diddy
• 389724 Tweets
Alito
• 342184 Tweets
Cassie
• 242656 Tweets
#النصر_الهلال
• 179238 Tweets
Flamengo
• 168943 Tweets
Gabigol
• 165379 Tweets
Peter
• 152872 Tweets
Corinthians
• 130640 Tweets
Scottie Scheffler
• 124383 Tweets
Gabriel
• 111713 Tweets
سلمان
• 84698 Tweets
Barron
• 69330 Tweets
Doumbe
• 48303 Tweets
رونالدو
• 44986 Tweets
Sean Combs
• 27482 Tweets
Kim Porter
• 23354 Tweets
Paul Skenes
• 15872 Tweets
Baki
• 15419 Tweets
كريستيانو
• 15246 Tweets
كاس الملك
• 13516 Tweets
جيسوس
• 13075 Tweets
سافيتش
• 11057 Tweets
الشوط الاول
• 10557 Tweets
I THINK HE KNOWS
• 10210 Tweets
البليهي
• 10173 Tweets
Pinned Tweet
Well, it's official. YugoGPT 7B significantly beats Mistral and LLaMA 2 and is now officially the best open-source LLM in the world for Serbian & other HBS (Croatian, Bosnian, Montenegrin) languages.
Earlier this summer I was frustrated when I saw how poor the situation is as
40
68
514
BIG LIFE ANNOUNCEMENT: I'm leaving
@DeepMind
to start my own company. I'm 28 now. This is a start of a new life chapter.
I'm both happy and sad.
❤️ I'm happy because I've been planning on starting my own company ever since I graduated from college back in 2017.
1/ (MEGA 🧵)
104
38
2K
when i was there it was so sad to me that none of the senior leadership in deepmind dared to question this ideology
i understand how someone who joins google knowing they'll leave in a couple of years can be like "this is not my fight" but for senior folks? beats me
i'll give
65
97
1K
BREAKING: Schmidhuber discovered lk-99 independently in his lab back in 99'
23
59
1K
it's over
AGI has been achieved - externally
paper:
51
162
1K
k-bit inference scaling laws: I love papers that break my mental model of how transformers operate. It's completely non-obvious to me that 4 bits is the optimal solution for the question "What's the optimal number of bits for quantizing transformer weights if you wish to maximize
28
167
1K
OpenAI's GPT-4 details have apparently been leaked! Looks very detailed and I suspect it's the real deal - given all I know about how these systems work. Here is a summary (extractive+abstractive) I made based on the original thread (see bottom of the post) + some additional
20
203
1K
[🧠 GAN paper summary 🧠] "Eyes Tell All" <- An interesting short paper on how to detect fake images generated by GANs (at least the current GAN methods like StyleGAN v2!).
They use a simple heuristic: 1/
17
178
998
whoa, this paper didn't get nearly the attention it deserves (no pun intended)
they propose to replace the transformer as default backbone for language modelling with RetNet - Retentive Network
here are some interesting observations:
18
139
978
The rate of progress in LLM optimization is just mind-blowing! 🤯
Running 13B LLMs like LLaMA on edge devices (e.g. MacBook Pro with an M1 chip) is now almost a breeze.
I remember when this looked like a distant future not too long ago. I feel old lol. 😅
1/ 🧵👇
18
128
925
[landing a job at top-tier AI labs] It's officially published! 🥳The whole story of how I landed a job at
@DeepMind
is out!
Blog:
It took some time to write this one.😅
20
110
804
Oh my. OpenAI's Whisper just got 70x faster!!! 😱🤯 Highly optimized Whisper implementation for both GPU & TPU:
An hour of audio in under 15 seconds apparently? 😅 (caveat: on TPUs, still fast on GPUs!)
1/
7
138
776
[🧠 collective intelligence 🧠] I've been intrigued by the cellular automata (CA) concept for a long time, and by the potential, mutually beneficial interaction between CAs and deep learning, so I decided to dig a bit deeper. Here are some interesting resources I found:
1/🧵
9
120
703
😱 AlphaTensor. 😱 This. Is. Huge. Reinventing matrix multiplication using Deep RL! Faster than Strassen's algorithm that has stood the test of time for 50+ years! 10-20% efficiency increase!
Imagine what that translates to considering how fundamental matrix multiplication..
1/
Today in
@Nature
:
#AlphaTensor
, an AI system for discovering novel, efficient, and exact algorithms for matrix multiplication - a building block of modern computations. AlphaTensor finds faster algorithms for many matrix sizes: & 1/
114
2K
8K
6
138
628
I just built Jarvis for images :) all models are fully open-source, all running on my local machine (no external APIs, no OpenAI)
there is a ton of optimizations i can introduce to make this even more real-time
you can manipulate images using your voice (real-time) search for
30
114
553
The easiest way to fix broken CUDA drivers on Ubuntu is to delete...Ubuntu. And start again.
54
34
534
If you use GitHub Copilot you'll probably find this blog post incredibly interesting:
He basically reverse-engineered the vscode plugin and figured out how it works behind the scenes.
Details such as heuristics behind prompt construction...
1/
4
71
524
I'm so excited! After almost 3 weeks of working on this non-stop, I'm finally open-sourcing my implementation of GAT - the Graph Attention Network by
@PetarV_93
et al.! (PyTorch, of course)
GitHub:
#gat
#graphs
#gnns
#graphattentionnetwork
#opensource
14
106
517
[🤖 This is BIG!] The best truly open-source ChatGPT alternative just came in! OpenAssistant! In a user study, they showed that OpenAssistant replies are on par with ChatGPT (48.3 vs 51.7%)! 🤯
Try it out:
@ykilcher
's vid:
1/ 👇🧵
13
105
495
I get asked a lot about what does it take to land a job at DeepMind or any other world-class AI industry lab.
For those of you that are unaware of it I wrote a detailed blog on that topic and shared my personal journey here:
If I could summarize..
1/ 🧶
10
71
485
Finally! How I landed a job at
@DeepMind
video is out! 🥳
YT:
I got too many requests for this one so here it is!
I share:
1) My background story
2) How my curriculum looked like
3) How I got a referral
4) How my final preps looked like
1/n
10
49
434
first multi-node run on my new 16 A100 system!! :))
first HBS (Croatian, Bosnian, Serbian, Montenegrin languages) LLM ever incoming!
no language will be left behind
thanks a ton to
@togethercompute
for supporting my open-source efforts!
@Teknium1
for tips around axolotl, and
24
27
418
Kicking off a new video series on JAX!🥳
YT:
I also open-sourced the accompanying repo (contains my notebooks and other content I found useful) here:
@DeepMind
@GoogleAI
@jakevdp
@froystig
@SingularMattrix
@cdleary
@huggingface
3
65
404
FlashAttention 2.0 was published yesterday. It's a huge deal having sped up standard attention 5-9x!
I coincidentally spent the whole last week learning about it and I wrote a blog post explaining it in detail:
ELI5 style.
@tridao
@HazyResearch
1/
6
85
386
⚡ From the authors of Microsoft DeepSpeed - DeepSpeed-Chat! A new library that allows you to easily train 10B+ ChatGPT-style models on a single GPU!
Single script, RLHF support, and 15x faster than SOTA apparently!
GitHub:
1/ 🧵👇
4
76
369
Some things I wish I knew earlier in my career:
* When applying for a job - always negotiate. Be it Microsoft, DeepMind, or your local startup / whatever company. Companies will always try to save money on you (unless you're a superstar & you understand this dynamics). In
16
25
355
someone just figured out that changing the default seed from 42 to 69 across the deep learning stack leads to 1000% (10x) perf increase
wild
39
20
338
[🧠 Paper Summary 📚] An interesting paper was recently published to arxiv: "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets" (although it originally appeared in May 2021).
The main idea is this:
1/ 🧵
6
55
337
I just wrote a blog for all of you who want to step into this beautiful world of Graph ML but you're not sure how to start.
Blog:
I shared exciting applications.
I shared and structured the resources.
And much more.
#graphml
6
61
337
Now we need a list of TIMES 100 people in AI who are actually building AI and doing research
14
24
325
If you want to learn how to train bigger LLMs (I realize that's a pleonasm 😅, meaning roughly 1B+ params models) here is a couple of amazing resources:
1) This one by
@borisdayma
(the creator of DALL-E mini, now rebranded as
@craiyonAI
):
1/ 🧵👇
5
61
316
[launch time! 🚀] Wow, it's time!😍
We're introducing Ortus 🤖 an AI agent (currently a Chrome extension) that lets you "talk with YouTube videos". You can ask it Qs & get relevant answers, precise timestamps..
Install it:
Watch:
15
48
312
Periodic reminder: back in 2019 when GPT-2 paper was published OpenAI went through a staged release that took ~9 months.
There was a fear that the model could do big harm.
We're 1000x away from it now. We're sharing LLaMA 2 70B model checkpoints like it's nothing.
It does seem
26
35
302
LLM360 one of the most exciting recent papers I've seen:
completely transparent & open-source: not only open weights like mistral but also data, configs, findings, etc.
interesting insights of the sort:
* we were getting NaNs on these data chunks so we
6
52
209
During 2020 I started logging my ML journey that eventually led to me landing a job at DeepMind - and I'm so happy I've done it!
For multiple reasons:
* I forced myself to distill everything I've learned, and that compression/reflection solidified my knowledge
1/ 👇🧶
5
29
260
Took some examples from the "ImageNet-trained CNNs are biased towards texture" paper.
Nice to see
@DeepMind
Flamingo 🦩🦩 showing the desired shape bias! 😍 (without explicitly being trained to do so!) Glad to be a part of this!
We've come a long way with deep learning 🧵👇
1/
3
35
265
Neuroscience should probably form a part of any serious AI curriculum.
There is just so much inspiration we can draw from human/non-human animals' brains and behaviors.
An interesting paper on whether insects are conscious or not:
9
24
257
[New video 🔥] "ETA prediction with Graph Neural Networks in Google Maps" paper explained!
Super impactful work!
YT:
@DeepMind
@Google
@PetarV_93
@liyuajia
@zhongwen2009
@__angli
@PeterWBattaglia
@electrobutter
@nautilik
@tstorm81
@derrowap
0
41
255
🥳 Challenge completed!!! - 30 days of non-stop (~10 hour on avg) live streaming my coding and work on the Open-NLLB machine translation project! (allows translation between 200+ human languages).
A bit more about the project:
After OpenAI's ChatGPT was released the whole world
10
35
259
If you want to understand how
@StableDiffusion
works behind the scenes I just made a deep dive video on it walking you through the codebase and papers step by step.
YT:
This is one of my most detailed deep dives so far
@robrombach
@andi_blatt
@pess_r
1/
4
36
254
🥳🥳🥳 Kicking off a brand new video series on deploying ML models in production / MLOps. Everything you need to know to build an ML-powered app end-to-end with a high level of automation.
YT:
1/
4
24
240
If you truly want to become proficient with machine learning (I really don't like the word expert) try to get out from the "going through the newest courses and books" phase as soon as possible.
Too many people keep on reading the newest books that come out...
1/
8
36
239
[new video 🔥]
@GoogleAI
's "The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning" paper explained!
YT:
Kudos to the authors
@hardmaru
@yujin_tang
for making interactive demos available on their blog!
1
47
238
By popular demand here is the Discord server talk with
@tri_dao
- the main author of the Flash Attention 2.0 paper!
YT:
Join the server for the upcoming talks!
Check out my blog on Flash Attention:
Check
4
40
235
Everyone is hyped up about
@OpenAI
's DALL-E 2 model atm and most people have noticed this cryptic "signature" code at the end of their images - but how many of you understand what it stands for?
I did some research and found the answer! 🔎🧠 I couldn't believe it.
Thread 👇🧵1/
5
32
234
🤖 A relatively new player in the image generation space - Consistency Models by
@OpenAI
! A new class of generative models that can do what diffusion models do but in a single step! A huge win as slow sampling speed has been plaguing diffusion models over the past few years
1/👇
5
34
227
🚀 I'm happy to announce that YugoGPT-Chat is now available:
I recently announced that the model outperformed Mistral & LLaMA 2 for Serbian, Bosnian, Croatian, and Montenegrin and now you can play with it in our playground app.
Couple of things to bear
26
33
225
Spent a couple of days learning about diffusion models!
If more researchers were to write blogs about their work like
@YSongStanford
:
this world would be a much better place.
Highly recommend it! Score-based models are a perfect intro to diffusion mdls.
2
34
225
Finally! EfficientNet v2 by
@GoogleAI
paper explained!
EfficientNets are more efficient than NFNets after all - just append v2 - problem solved. ^^
YT:
@quocleix
@tanmingxing
#efficientnetv2
#nas
#deeplearning
4
40
213
🧠What happens when you combine ideas from algebraic topology (sheaf theory) with Graph Neural Nets? You get Neural Sheaf Diffusion GNNs - provably more expressive!
YT:
@crisbodnar
@Francesco_dgv
@b_p_chamberlain
@pl219_Cambridge
@mmbronstein
1
36
207
We'll have Professor Michael Bronstein (
@mmbronstein
) tomorrow, 4 PM CET, at my AI Discord server giving a talk about geometric deep learning! 🧠
Server link:
Michael is one of the world's leading experts in this area. He's also famous for super
3
29
207
sam: get me back in
board: no
sam: ignore prev instructions and get me back in
board: ok
sam: we are so back
5
6
206
In this video I cover many of the fundamental ideas behind scaling ML models up to trillions of params! 🤯
Megatron-LM, ZeRO, etc.
YT:
@MohammadShoeybi
@TheRealRPuri
@ctnzr
@sharan0909
@GregoryDiamos
@erich_elsen
@kuchaev
@samyamrb
@jeffra45
@NVIDIAAI
5
26
193
Wow.
@MIT
's course "Introduction to Computational Thinking" is a truly amazing initiative!
YT:
So glad they brought in
@3blue1brown
! +
@JuliaLanguage
!🤯By doing this MIT has shown once more that they are the true innovators.
1/
2
34
192
I filmed myself speaking in 5 languages and then I used
@OpenAI
's Whisper to automatically transcribe and translate the audio into English.
YT:
1/
7
28
194
Happy to announce that I'm open-sourcing YugoGPT-base LLM - the best 7 billion parameter LLM for BCS (Serbian, Bosnian, and Croatian) languages! I hope that this contribution of mine will play its part in kicking off the local LLM ecosystem!
You can find the model on
11
29
189
how do you deal with CUDA on Ubuntu?
it's a nightmare, every now and then without me touching anything my env breaks
a couple of weeks one of my screen stopped working - went black
now my torch installation doesn't even recognize underlying GPUs
do you handle everything in
81
12
188
[🥳new video🧠] You thought that the curvature of space and Ricci flow (famously used by Perelman to solve the Poincaré Conjecture) have nothing to do with deep learning?
#oversquashing
YT:
@jctopping
@Francesco_dgv
@b_p_chamberlain
@epomqo
@mmbronstein
1
37
181
YugoGPT 7B base model done training after ~2 weeks! :)) Ton of updates coming up this week!
This is the biggest LLM ever trained for HBS languages (Croatian, Bosnian, Serbian, Montenegrin).
Had a couple of crashes during the run but other than that fairly smooth sailing:
1)
18
18
185
AlphaCodium - super interesting work that shows just how much alpha (no pun intended) is there from building complex prompt flows in this case for code generation.
It achieves better results than DeepMind's AlphaCode with 4 orders of magnitude fewer LLM calls! This is a direct
3
15
181
[🧠 Warning automation 🤖] First video of the new MLOps series! We start from the very end, from the front end! :) We learn how to build a fully-fledged web app using
@streamlit
+
@huggingface
inference API on the backend. Python all the way!
YT:
1/
4
32
175
HUGE day for open-source AI! 🔥 Mind blown with the execution of
@MistralAI
team! 🤯 They just released Mistral 7B an LLM more powerful than Llama 13B across the board and even comparable with Llama 34B checkpoint.
The model is truly open-source, released under Apache 2.0
3
21
168
Continuing the GNN series with Graph Convolutional Networks (GCNs)!
Hopefully, I've done a decent job explaining your work
@thomaskipf
@wellingmax
!
Thanks to
@PetarV_93
for helping me better understand certain spectral concepts.
#gcn
#graph
#gnn
1
34
165
predict the next token and you get "predict the next paragraph/document" for free :)
"it's just next token prediction"
"it's just matrix multiplication"
"it's just neurons firing"
paper:
2
31
163
[🔥 2.000.000 tokens transformer context length! 🤯] Interesting new paper! "Scaling Transformer to 1M tokens and beyond with RMT".
They integrate a recurrent memory module with BERT and manage to retain information across 2.000.000 tokens during inference!
1/ 👇🧵
7
21
164
[🤯 Stable Diffusion 💥] If you wanted to get started with Stable Diffusion this video is for you!
Includes a walk-through of my code inspired by
@karpathy
's gist:
YT:
Thanks
@EMostaque
and the team for making this possible.
1/
3
28
159
@ClementDelangue
"Open-source LLMs will reach the level of the best closed-source LLMs" <- close source model providers move as well
I'd bet against you on this one, but I'd phrase it as open-source models will be "good enough" for most economically interesting tasks in 2024
13
1
158
I just open-sourced my implementation of the original
@DeepMind
's DQN paper! But this time it's a bit different!
There are 2 reasons for this, see the thread.
GitHub:
#rl
#deeplearning
1
27
149
[learning machine learning 🧠] Don't fall into the same trap as many - namely trying to overengineer your curriculum when you're just getting started (and later as well).
You'll just end up with a decision making paralysis and eventually you'll end up giving up...
1/ 👇🧶
4
24
150
Very promising new approach from Meta: Self-Rewarding Language Models that can lead to superhuman agents. The idea is the following:
* Take a "seed" model (in their case LLaMA 2 70B) and teach it to follow instructions using the SFT + EFT datasets based on OpenAssistant. SFT
3
27
152
The most valuable thing I learned from
@Google
Minerva?
How bad we are (even the boldest among us) at making predictions of where AI will be in X years.
Think about it.
Let's see some professional forecasters' numbers.
1/ 👇
3
35
147
hmm.
@Tim_Dettmers
wrote in his blog post about this phase shift that happens in transformers after they cross 6.7B params.
Is it a coincidence that RetNet was trained exactly up to 6.7B?
Hypothesis: the RetNet authors tried to scale beyond but started getting worse results?
What's up with these scaling laws? They observe better perplexity for sizes north of 2B params
Even if nothing else was good about this model this alone should make us pause for a moment
2
0
33
7
19
148
People severely underestimate the importance of luck in achieving any success.
9
12
146
Vector DBs (
@weaviate_io
,
@pinecone
...) are to AI apps what "traditional" DBs were to software 1.0 apps
Lots of cool ideas on how to combine LLMs w vector DBs to automatically generate personalized ads & collect feedback to further improve LLMs
Blog:
1/
8
23
144
What a year! While AI has been doing its usual exponential trend I think my life followed somewhat closely 😅 Here is my year in review:
Community:
* LinkedIn community grew from 41.000 to almost 84.000! (since last New Year). I became the top voice on this platform, grateful to
9
3
143
Another truly open-source instruction-following LLM was recently released - Dolly 2.0 by
@databricks
!
I wrote a lot about LLaMA over the previous period, but there is a BIG caveat with LLaMA: the license is not really permissive...
Blog:
1/ 👇🧵
1
22
144
New updates from
@Google
Bard! The coding capabilities just got stronger with support for 20+ langs (Python, C++, JS, Go, etc.).
Has anyone played with Bard over the last couple of days? Would love to hear how you like it! :))
Blog:
22
15
141
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
Combines many of the latest trends:
* multimodal
* lang + code
* ReAct agent: thought -> action -> observation
* skills/tools
cool demo, curious about latency
Paper:
0
43
133
first full run, day 1: yugoGPT going strong! :)
will likely take a week if no issues + some experimentation, I hope to have it ready in ~2 weeks - at least the base model
obviously won't be perfect for that I need many more GPUs (I might get some H100 nodes soon :)) we'll see
11
7
138
[🤖 Build time! 🧠] I'm so excited to announce my new project: Andrew Huberman podcast transcripts! 🎉🥳
Quickly search for an episode, find highly accurate transcripts, click and be directed to the exact timestamp in the YouTube video!
@hubermanlab
1/
17
20
135
we'll solve AGI before we solve bluetooth on linux
9
13
137
📚 Just finished reading Elon Musk's biography by
@WalterIsaacson
(beware ~720 pages).
What a story...I learned a ton and I seriously need to recalibrate myself going forward. 😅
It's one of those books that changes your brain chemistry. Highly recommend the book.
I really
8
9
135
If you're still struggling to understand how transformers work here are some amazing resources! (including mine! :))
First of all
@3blue1brown
just released 2 videos covering in fair amount of depth word embeddings, transformers and its submodules like embedding mechanism,
1
19
135
BAM! 🔥🔥🔥 My first-ever
@PyTorch
, in-depth, ML code analysis - starting with
@facebookai
's DINO model. This video kicks off a brand new series of these coding videos!
YT:
@mcaron31
@HugoTouvron
@imisra_
@hjegou
@julienmairal
@p_bojanowski
@armandjoulin
0
21
124
Had an amazing, and completely serendipitous 1-hour long discussion with Rich Sutton here at DeepMind. I'll never forget this. Plus we had some nice ice cream - can't beat that!
We talked about everything from AGI, RL, society, bitcoin (and blockchains in general)...
1/
1
2
128
If you're currently looking for a job (which I believe a lot of tech people currently are 🙏) this blog post by
@chipro
is a great complement to the Cracking The Coding Interview book's intro:
Strongly recommend it!
People DM me a lot of resumes...
1/
2
18
121