Simo Ryu
@cloneofsimo
Followers
14K
Following
2K
Media
1K
Statuses
4K
Will be at sf for a week or so from july 22~. Hmu if u wana chat on whatever interesting.
0
0
28
POV : you have access to sql database.UPDATE FAL_USER_CREDIT.SET CREDIT = CREDIT + 10000.WHERE USERNAME = 'cloneofsimo';
1
1
29
MuonClip. so many tricks to make maximum logits bounded during training. Gets me wondering why dont people try LASER (and maybe, z-loss ?).
Very interesting, standard attention causes vanishing gradient due to most prob being very small after some training. LASER tackles this by pushing the attention operation on exponential space. i.e., exp_output = sm(QK^T) exp(V). They dont seem to exaggerate on the performance
7
21
303
Huge respect to Kimi for calling this optimizer after muon instead of re-branding into completely different name bullshit like all the other companies / academics do
🚀 Hello, Kimi K2! Open-Source Agentic Model!.🔹 1T total / 32B active MoE model.🔹 SOTA on SWE Bench Verified, Tau2 & AceBench among open models.🔹Strong in coding and agentic tasks.🐤 Multimodal & thought-mode not supported for now. With Kimi K2, advanced agentic intelligence
9
40
760
Unironically best thing ive ever done in my life was poaching @hanchchch .Hes barely on twitter but if there is 10xEng he is the one. @FAL just keeps winning.
3
0
49
This is incredible! muP allows you to predict transfer across width beyond typical noise level.
While scaling laws typically predict the final loss, we show in our ICML oral paper that good scaling rules enable accurate predictions of entire loss curves of larger models from smaller ones!. w/@Locchiu, @andrewgwils, J. Pennington, A. Agarwala:.1/10
2
0
45
Matformer and MoE is like mutually exclusive, but both somehow reasonable.How do we combine both idea into reality? Have someone tried matformer within each expert?.
3
0
30
RT @ezyang: Want to play around with code that uses PyTorch DTensors but annoyed at having to go multiprocess? Make a fake process group! T….
0
26
0
all reasonable things sit on mass = radius^3 plot (as it should be lmao). except for black holes, and dying stars
0
0
7
"All objects and some question".by Charles H. Lineweaver and Vihan M. Patel. We sit at the dead center of this plot, makes me feel so small but also so important
1
0
14
The very fact that this post went viral is the hint that the bro's X-Bible is very powerful indeed.
I broke down Twitter's entire algorithm repo w/ Claude Code. And built "The X Bible" tool that scrapes ur profile+feed and identifies profile misalignment and how to go viral in ur niche. comment "X Bible" and I'll give u access to the tool. only sharing with first 200 for rn
1
0
14
if your LLM is trying to argue with you, just say these powerful words
1
3
41
