Jonathon Belotti
@jonobelotti_IO
Followers
892
Following
4K
Media
33
Statuses
245
Peeling back the layers @modal. Previously ML platform @canva. dms open.
NYC
Joined April 2016
Kinda crazy that you can now turn a live container and its GPUs into a couple files on disk. The cold starts are mad fast.
We just launched GPU memory snapshotting on @modal_labs in alpha. Speed up cold boots by up to 12x 😇. If you're deploying AI models, a huge amount of cold boot time comes from loading model weights into GPU memory. This makes it difficult to scale GPU resources up and down
1
0
30
A goldmine of performance data for those building AI systems. Or, more fittingly, a seed bank.
Dozens of teams have asked my advice on running LLMs. How fast is @deepseek_ai V3 with vLLM on 8 GPUs? What's the max throughput of @Alibaba_Qwen 2.5 Coder with SGLang on one H100?. Running & sharing benchmarks ad hoc was too slow. So we built a tiny app, the LLM Engine Advisor
0
1
12
RT @charles_irl: fun fact, this is only the second time i have seen Colin smiling while thinking about GPUs. every other time he wore the p….
0
2
0
When I was at Canva I had to get a batch job to run demographics analysis vision models on over 90 million images. It took a couple weeks, but felt like with the right infra it could be done in a couple hours. The infra now exists :).
Today, we're launching Modal Batch. Run large-scale batch jobs that scale across thousands of containers without writing a single line of YAML.
0
2
58
2/2 Modal has an increasingly detailed model of demand and supply, but it can almost all boil down into a linear solve sitting at the centre of our capacity management system.
1
0
19
1/2 Did you know a few months ago you could get hundreds of superior H200 GPUs for 20% less than the going rate for H100s? Our capacity solver did, and it took that deal. By automating price, capacity, and reliability data collection across all hyperscalers, we can feed a linear.
1
1
23
Today on the blog, we won't tell you where the deep and cheap GPU capacity is, but we will teach you how to fish 🎣.
7
21
300
You can't stop @ekzhang1 from shipping. You can't.
modal.com
Today we're releasing lightweight client libraries for JavaScript and Go, making it easier to start sandboxes and call serverless functions — no Python required.
7
6
118
Our developer advocate reads 2x more systems textbooks than the systems engineers. If the LLMs don't catch up with us, Charles will.
And the end of all our exploring.Will be to arrive where we started.And know the place for the first time
2
0
26
RT @kanjun: Today, AI can generate tons of code—but how do we know if it's good?. That's why we built Sculptor: the first coding agent envi….
0
87
0
RT @jeremyphoward: Wow someone has actually done some AI-archeology to attempt to answer the question "what was the first LLM"! :D. It's a….
0
15
0
Looking forward to doing a drive by next week
1
0
6
Thanks @ekzhang1 for continuing to run a wonderful little group. cool presentation today.
thanks @chenyuxyz for the awesome talk the internals of tinygrad today at nysrg! you can tell when you’re talking to a real expert, and having chenyu as an expert on kernels & hardware optimization come hang out with us for 3 hours was really wonderful
0
1
11
RT @charles_irl: we're hiring at @modal_labs!. this is, without exaggeration, the most cracked team i have ever worked with, and i firmly b….
0
7
0
At @modal_labs your customer questions sometimes get a whole blog post :). Why does an NVIDIA H100 SXM 80GB card offer 85.52 GB?
1
0
13
Good day to be a container snapshots fan. CUDA snapshotting is finally released.
2
2
24
Why is the restore so much faster than our standard container startup? That answer is too much for a thread. See the full blog post here:
modal.com
Serializing container state to disk for aggressive cold start optimization.
0
0
18