From GPT to MoE: I reviewed & compared the main LLMs of 2025 in terms of their architectural design from DeepSeek-V3 to Kimi 2. Multi-head Latent Attention, sliding window attention, new Post- & Pre-Norm placements, NoPE, shared-expert MoEs, and more.

Next to Qwen3 of comparable size: Looks like gpt-oss is a wide (vs deep) model

Finally! Will be working through the internals in more detail. The first surprising fun fact is they used bias units in the attention mechanism like ye goode old GPT-2. Super interesting, I haven't seen any other architecture doing that since then!

Link to the standalone-qwen3-moe.ipynb notebook: And a variant with KV cache (standalone-qwen3-moe-plus-kvcache.ipynb):

So, I did some coding this week. - Qwen3 Coder Flash (30B-A3B).- Mixture-of-Experts setup with 128 experts, 8 active per token.- In pure PyTorch (optimized for human readability).- in a standalone Jupyter notebook.- Runs on a single A100

And let’s not forget that this model can run locally so you have to worry less about having private code bases accessed by proprietary apps and vendors.

This might be the best coding model yet. General-purpose is cool, but if you want the best at coding, specialization wins. No free lunch.

The new Qwen3 update takes back the benchmark crown from Kimi 2. Some highlights of how Qwen3 235B-A22B differs from Kimi 2: .- 4.25x smaller overall but has more layers (transformer blocks); 235B vs 1 trillion.- 1.5x fewer active parameters (22B vs. 32B).- much fewer experts in

Btw if you're learning how to build LLMs from the ground up, there's now a 17h companion video course for my LLMs From Scratch book on Manning: It follows the book chapter by chapter, so it works great either as a standalone or code-along resource. It's

Want to clarify that I only focused on the architectures there. Yes, one key novelty is not well reflected in the architecture but in the way it was trained. I.e., they replaced the popular AdamW optimizer with a modified version of Muon. According to the Kimi 2 developers,.

Kimi K2 is basically DeepSeek V3 but with fewer heads and more experts:

RT @Yuchenj_UW: Holy shit. Kimi K2 was pre-trained on 15.5T tokens using MuonClip with zero training spike. Muon has officially scaled to….

RT @rasbt: @Sentdex Right now LLM providers serve you one-size-fits-all LLMs, which is one of the main bottlenecks.

</sarcasm>.

TIL Cursor was literally named by the devs after all the cursing it caused when it trashed their own codebase.

RT @ShaneAParrish: I started coding again recently. No, not vibe coding. Actual coding. I fell in love with coding as a teenager. I’d st….

There were many people who asked me about joining their startup & co-founding a startup in the past 2 weeks. Not career advice, but just my 2 cents:.I actually think it’s a great time to start a bootstrapped startup compared to going the VC-backed route. These days, there are.

Surprised that no one mentioned the lack of a table of contents. In any case, there is one now!

If you're getting into LLMs, PyTorch is essential. And lot of folks asked for beginner-friendly material, so I put this together:.PyTorch in One Hour: From Tensors to Multi-GPU Training (.📖 ~1h to read through.💡 Maybe the perfect weekend project!?. I’ve.

Updated the notebook to support the other dense models as well: Qwen3 0.6B, 1.7B, 4B, 8B, and 32B.