LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels. Full blog: Evals, efficiency, and more ↓.

Excited to announce we’re building an Applied Team focused on post-training. Come explore what's possible with our new (and still unreleased) LTM2 models and their 100M token context window. Apply here:

Very excited to welcome @nvidia as Magic's latest investor! With their support, we’re looking forward to scaling long context and inference-time compute.

With context solved, we now focus on unbounded inference-time compute as the next (and potentially last) breakthrough we believe is needed to build reliable AGI. Imagine if you could spend $100 and 10 minutes on one task and reliably get a great pull request for an entire.

Our LTM (Long Term Memory) mechanism needs >1,000x less compute and memory than Llama 3.1 405B’s attention. Llama 3.1 would need 638 H100s *per user* to store a 100M token KV cache. LTM needs a small fraction of one. SSMs, RNNs, and RAG all exploit weaknesses in evals like

RT @EricSteinb: Very excited to welcome @karpathy as Magic's latest investor!.

RT @Hersh_Desai: The era of long context is upon us. The question is whether you want to be 1 of 1000 co-authors on the Gemini paper or 1 o….

RT @Hersh_Desai: I have been continuously in awe of the brilliance, tenacity, and kindness of @EricSteinb and the small but mighty team at….

RT @EricSteinb: I love my team a lot and sometimes it’s stressful but life has never been so fulfilling. If you want to build AGI on a smal….

If you want to solve very hard problems to build safe AGI on a small team with thousands of GPUs, come join us: .

This round was led by @natfriedman & @danielgross with participation from @CapitalG and @eladgil, and will allow us to further scale up our models.

We've raised $117M from @natfriedman and others to build an AI software engineer. Code generation is both a product and a path to AGI, requiring new algorithms, lots of CUDA, frontier-scale training, RL, and a new UI. We are hiring!

RT @goodside: 5M tokens of context. Let that sink in. Yes, there's caveats. But consider what's to come:.- Entire codebases in prompts.- N….

RT @natfriedman: 👀 showing a sneak peak of their 5M token context code model.

RT @EricSteinb: AI with long-term memory!. *A lot* of work left to do but happy to share a little more about what we've been up to. It's b….

Turn old photos into videos and see friends and family come to life. Try Grok Imagine, free for a limited time.

What’s next? More compute. LTM Nets see more context than GPTs, but LTM-1 has fewer parameters than today’s frontier models, making it less smart. Knowing how drastically model scale improves the performance of GPTs, we're excited to see how far we can take LTM Nets.

How?. We tried to scale standard GPT context windows but quickly got stuck. So, we designed a new approach: the Long-term Memory Network (LTM Net). Training and serving LTM Nets required a custom ML stack, from GPU kernels to how we distribute the model across a cluster.

Watch LTM-1 reuse and synthesize information across files:

Watch LTM-1 generate complex suggestions:

Meet LTM-1: LLM with *5,000,000 prompt tokens*. That's ~500k lines of code or ~5k files, enough to fully cover most repositories. LTM-1 is a prototype of a neural network architecture we designed for giant context windows.