RT @rdyro128523: Llama 4 inference in pure JAX! Expert/tensor parallelism with int8 quantization. Contributions welcome! .

A nice and concise R1 inference jax:tpu port by @rdyro128523. Good for both reading and running. Watch the repo for more.

RT @JeffDean: Training our most capable Gemini models relies heavily on our JAX software stack + Google's TPU hardware platforms. If you….

Our online book on systems principles of LLM scaling is live. We hope that it helps you make the most of your computing resources. Enjoy!.

RT @sharadvikram: We now have a guide to writing distributed communication on TPU using Pallas, written by @JustinFu769512! .

RT @jxbz: Modula x JAX = Modulax. @gallabytes is cracked and ported Modula into JAX in a few days. I haven't had a chance to test yet, but….

RT @exoplaneteer: I've finally landed my first proper JAX feature since joining the team: a supported "foreign function interface", which m….

RT @sharadvikram: Finally got around to writing a guide for matrix multiplication on TPUs using Pallas. Check it out!. .

RT @apaszke: Many of you are excited about H100 attention, so it’s a good time to show you Mosaic GPU: a Python DSL for H100s. The attenti….

RT @_ddjohnson: Excited to share Penzai, a JAX research toolkit from @GoogleDeepMind for building, editing, and visualizing neural networks….

RT @sharadvikram: Built with JAX!.

RT @dlwh: Today, I’m excited to announce the release of Levanter 1.0, our new JAX-based framework for training foundation models, which we’….

There's a longer history in PL of thinking about linearity in programs, what it means, and what we can do with it (cf. linear types/logic). Hopefully distilling a big piece of AD down to linearizing stuff makes it easier to think about what programs we can differentiate, and how.

To get there, we need to identify linearity *in programs*. For functions expressed in code, we want a notion of linearity that implies mathematical linearity, but that also allows a compiler to delineate and transpose things automatically.

Composing the two, we get reverse-mode AD as you know it today. The implementation is simpler, disentangling perturbation from reversal.

We turn these algebraic facts into algorithms: "linearization" amounts to extracting the linear computation from forward-mode. "Transposition" roughly means reversing that extracted program. These steps can be written separately, like compiler passes.

A key concept is *linearity*. Differentiation forms a linear map—the Jacobian. Forward-mode AD computes that map (aka the "JVP", for Jacobian-vector product). Reverse-mode computes its transpose ("VJP").

We've always done AD this way, and we wrote about it briefly before (. The new paper tries to go into more detail by working over a minimal programming language.

In JAX and Dex, we do automatic differentiation (AD) in a distinctive way: by "linearizing" and then "transposing" programs. We wrote up what this looks like in a model language: with Alexey Radul, @apaszke, @SingularMattrix, @DougalMaclaurin.

RT @mblondel_ml: After 6 months of hard work, happy to share JAXopt: hardware accelerated, batchable and differentiable optimizers in JAX h….