The Monarch release from PyTorch is super neat, big fan of clean mesh abstractions. They're using it in their new RL framework, TorchForge.

Model conversion (like operator grafting or swapping the modeling objective here) is a key tool for architecture research nowadays, given the prohibitive cost of pretraining. Thrilled to see a lab focusing on fast feedback loops partially based on model conversions, excited to

It's a good model sir. Very proud of the team, we worked very hard to be on the Pareto frontier of quality and efficiency. Even had the chance to write a CPU-optimized kernel for MoE to squeeze everything from the hardware, and that gave us those sweet throughput results.

Source:

Found a short and clean walkthrough for how MMA on Tensor Cores works at the PTX level

Note that the argument is very similar to the findings by the 🐐 @ZeyuanAllenZhu which motivates its introduction of Canon layers (which are short conv1d), and proves that adding them dramatically improves GLA. (This is in Physics of LLM 4.1)

Full derivation:

Went through a lengthy derivation of MLA today to understand exactly how we could absorb the KV decompression matrices, as I didn't find much code on it. The trick is to write the proof on a per-head basis to easily derive the absorption. In the end, it does give an enlightening

Interesting take on the role of Short Conv. The argument uses the view that linear attention variants do a form of online learning (i.e.eTTT) over the KV pairs (k_1,v_1), ... (k_t,v_t), such the update to the state is S_t​=S_{t−1}​−η_t​*∇_{St−1​} ​L(f(S_{t−1}​;k_t​),v_t​),

With SoRA, Veo, SeedDance coming out, a critical issue in building video generation models is the lack of automatic and interpretable evaluation metrics or reward models. To power the development of video generation models, we built VideoScore2, the SoTA generative video metric

The Dreamer 4 paper is really nice to read. Really appreciate the ablations on how the combinations of learning objectives (shortcut, x-loss, etc.) & architectural tweaks (e.g. hybrid, register tokens) affect speed and quality.

A note on the diffusion learning objective I hadn't realized before and how tweaking it ensures rollout stability (this may be specific to the case where we use diffusion forcing?). In the Dreamer 4 paper, they predict the clean representation x1 instead of the velocity v = (x1

Look at the cool model we just released, it's super fast! :) One innovation is that it uses both cont. & disc. audio tokens to enable gen. w/o losing understanding capabilities.

banger just dropped

small but mighty

The secret sauce most definitely is in the data, given that the architecure is fairly standard: Qwen3 backbone + NaViT SigLip2 (i.e. it uses packed vision sequences). They use patch_size=16 and pixel_shuffle_scale_factor=2 in order to use few image tokens. A 256x256 image will

S/o @samsja19

Switched the config system for the experiments to pydantic-settings, and I've never felt better. I have more energy. My skin is clearer. My eye sight has improved.

Many people would benefit from learning tensors contraction. Most concepts in ML architecture can be simplified and abstracted through this lens. Things with different names are just contracting, batching, sharding on different dimensions e.g. TP vs CP, BatchNorm vs LayerNorm,