Patrick Kidger
@PatrickKidger
Followers
9,037
Following
193
Media
128
Statuses
1,050
🧪BioML @ 🧑💻Prev. Google X, Oxford. 📚Neural ODE textbook: 🤖Open JAX ecosystem:
Zürich
Joined July 2020
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#RightPlaceWrongPerson
• 633081 Tweets
namjoon
• 371950 Tweets
Doge
• 184213 Tweets
RCB FINISHED DHOBI
• 159151 Tweets
How Sweet
• 141025 Tweets
LOST OUT NOW
• 118066 Tweets
新時代の扉
• 72116 Tweets
Clancy
• 71602 Tweets
Kabosu
• 58193 Tweets
#ThinkDeeply
• 41268 Tweets
Hayırlı Cumalar
• 33898 Tweets
#يوم_الجمعه
• 28510 Tweets
メラルバ
• 25081 Tweets
夏アニメ化
• 24104 Tweets
エアコン
• 22101 Tweets
ウマ娘の映画
• 21531 Tweets
移民政策
• 20159 Tweets
スタライ
• 19808 Tweets
Pinned Tweet
⚡️ My PhD thesis is on arXiv! ⚡️
To quote my examiners it is "the textbook of neural differential equations" - across ordinary/controlled/stochastic diffeqs.
w/ unpublished material:
- generalised adjoint methods
- symbolic regression
- + more!
v🧵 1/n
28
212
2K
So I've made a library - "torchtyping" - for annotating a PyTorch Tensor's shape (dtype, names, layout, ...)
And at runtime it checks that the annotations are consistent and correct!
Bye-bye bugs! Say hello to enforced, clear documentation of your code.
21
230
1K
Aaaaannouncing: my first open-source library since I joined Google!
🔥sympy2jax🔥
Build your physics-informed model in SymPy, perform arbitrary symbolic manipulations on it, then convert it to JAX and train it via gradient descent!
17
131
1K
Huge life update: I have finished my PhD at Oxford, and I am joining Google X!🌟(
@TheTeamAtX
)
(Using REDACTED to tackle impossible problems like REDACTED and REDACTED.🕵️♂️)
Currently at Google London, but I'm the only X-er here. So DM me if you're local, I'd love to make friends!
38
10
827
🌟🌟My team at Google X is hiring!
We're looking for someone in {Proteins}∩{ML}, to apply Transformers/LLMs to protein design.
(We're doing some really really cool stuff.🤖)
DM me + apply here:
This could be you!
14
74
640
Use your thesis title as the prompt for
#DALLE3
.
Here goes: "On Neural Differential Equations"
It's beautiful!
Use your PhD thesis title as the prompt 🤓
Is it time to restart this trend? But now with
#DALLE3
?
(Here is mine: "Protein structure determination using evolutionary information")
45
45
450
21
52
605
As a bit of an experiment, I've made my next academic poster in HTML+CSS -- not LaTeX!
IMO the result looks very pleasant.
I much prefer working with CSS over LaTeX when trying to make things look pretty.
WDYT?
18
32
596
🌟 New blog post: Torch-to-JAX!
I made the jump PyTorch->JAX a couple of years ago. (JAX is just awesome for the kind of sciML I do!)
So for those of you making the same jump, I wrote a JAX quickstart guide for a PyTorch developer. :)
Link:
⚡️
10
85
559
"jaxtyping" is an amazing way to catch shape errors in your
#JAX
code. Just add type annotations!🔍
(GitHub: )
And as of the new version, it now supports the holy grail of shape-checking: symbolic expressions!🎉🎉
For example:
8
68
501
New paper: Neural Rough Differential Equations !
Greatly increase performance on long time series, by using the mathematics of rough path theory.
Accepted at
#ICML2021
!
🧵: 1/n
(including a lot about what makes RNNs work)
3
90
496
💐Happy to announce that our
#ICML2022
paper...
...
...nah just kidding it was rejected lol.
13
9
480
⭐️Announcing Diffrax!⭐️
Numerical differential equation solvers in
#JAX
.
Very efficient, and with oodles of fun features!
GitHub:
Docs:
Install: `pip install diffrax`
🧵 1/n
6
95
449
Announcing Equinox!
A JAX neural network library with
- a PyTorch-like class API for model building
- whilst *also* being functional (no stored state)
It leverages two tricks: *filtered transformations* and *callable PyTrees*.
1/n🧵
10
64
393
So I wrote a blog post!☀️
"JAX vs Julia (vs PyTorch)"
With a focus on using
#JAX
/
#julialang
in scientific computing + ML computing; covering their similarities (speed, functors, homoiconicity) and differences (introspection, documentation, code quality).
8
60
387
Double descent?
Pfft, in *real industry* we get 9-fold descent!
(seriously what the hell is this)
(this is my actual training run)
54
8
368
I am writing up my PhD thesis in mathematics.
_I need to cite Isaac Newton._
Which means: I am currently staring at this page trying to figure out how to turn it into BibTeX. :D
9
29
350
Announcing a new library for
#JAX
!
⭐jaxtyping⭐
Type annotations (+runtime checking!✅) for the shape/dtype of JAX arrays, with a beautiful concise syntax: f32["foo bar"] means dtype float32 and shape (foo, bar).
Also annotate PyTrees!
GitHub:
1/2
6
44
350
✨Life update - I have joined in Zurich!✨
We're a startup/scaleup on ML for protein design. (And about half ex-Googlers, haha!) The team here are some of the best at this of anyone in the world.
My official job title is "machine learning wizard"!
1/
19
9
330
🌟Announcing "Lineax" - our newest
#JAX
library! For fast linear solves and least squares.
GitHub:
* Fast compile time
* Fast runtime
* Efficient new algorithms (e.g. QR) + existing ones (GMRES, LU, SVD, ...)
* Support for general linear operators🔥
1/
6
43
330
🔥So I wrote a blog post!🔥
"How to achieve success in a machine learning PhD?"
(I get this question a lot.)
🌟And here's my suggested answer: know all this stuff.
🔗
WDYT? What else would you include on your JKS list?
12
51
322
Our 'The Symbiosis of Deep Learning and Differential Equations' workshop has been accepted for
#NeurIPS2021
!
Send us your work on data-driven dynamical systems, neural differential equations, solving PDEs with deep learning etc.
Tentative submission deadline Sept. 17.
4
50
307
Happy to announce that our work on neural differential equations + time series - Neural CDEs - got accepted w/ spotlight at NeurIPS!
11
42
303
Very excited to announce our
#NeurIPS2021
paper!
**Efficient and Accurate Gradients for Neural SDEs**
Paper:
Example:
IMO this is the best paper I've ever written.
A thread:
1/n 🧵
4
40
258
As someone working in "scientific machine learning", I've found that the Python-vs-Julia debate comes up quite a lot!
This is an interesting recent thread on the state of ML in Julia, in particular with reference to PyTorch, JAX etc.:
3
47
254
🔥 FYI: it seems to have snuck by that jaxtyping (GitHub: ) now supports PyTorch, TensorFlow NumPy etc. as well! 🔥
IMO it is now always the best way to add shape/dtype annotations to your arrays/tensors. (It doesn't even require JAX as a dependency!) ⚡️
3
27
253
Oh this is super neat.
IIUC: apply a U-Net to each frame; you get a hierarchy (multiple resolutions) of latent spaces. The time evolution of the video then corresponds to running a differential equation in each latent space.
FWIW I usually advertise neural diffeqs for scientific
Bridging principles between physics and AI will result in new ideas that work well. We present our work on neural CDEs and continuous-time (CT) U-Nets. Our ideas are inspired by
@PatrickKidger
's work. Find our paper here: and see 🧵for a quick summary.
5
28
171
3
32
249
Announcing: the second iteration of our workshop at
#NeurIPS2022
!🌋
"The Symbiosis of Deep Learning and Differential Equations"
Accepting 4-page papers on neural ODEs/SDEs/PDEs, score-based diffusions, numerical methods, software libraries etc!
Website:
1
52
244
Put together a micro-library for turning SymPy expressions into PyTorch Modules.
Symbols becomes inputs, and floats become trainable parameters. Train your SymPy expressions by gradient descent!
3
34
235
🌟Time for another blog post! :D🌟
"No more shape errors! Type annotations for the shape+dtype of tensors/arrays."
Link:
I think the audience for this one is nearly everyone who uses PyTorch / NumPy / JAX / TensorFlow.🤖
7
36
231
For anyone else also jumping ship LaTeX -> Typst.
I wrote a Typst plugin that generate figures dynamically, by running inline Python code inside Typst documents.🌟
GitHub:
(CC
@typstapp
@MilesCranmer
@jj_rader
)
3
23
228
Two of my open-source libraries simultaneously broke 1k GitHub stars!⭐️
In particular it's great to see how useful TorchTyping has been to so many folks... I wrote it purely to procrastinate from my PhD! 😀
4
9
223
Oh my gosh, there's finally a convincing-looking replacement for LaTeX.
If this ends up being as good as it looks then it will be Christmas come early.
In one day - one day! - of going open source, the Typst typesetting system passed 5,000 stars on Github.
If you ever needed evidence that there is a real hunger for a TeX replacement, this is it.
16
57
409
9
13
205
🎆Announcing Mkposters🎆
By popular demand! Write your posters in Markdown; style them with CSS.
Beautiful results with no LaTeX.
GitHub:
Install: "pip install mkposters"
Usage: "python -m mkposters your-poster"
Thoughts?
1/2
As a bit of an experiment, I've made my next academic poster in HTML+CSS -- not LaTeX!
IMO the result looks very pleasant.
I much prefer working with CSS over LaTeX when trying to make things look pretty.
WDYT?
18
32
596
6
26
200
So I just passed my thesis defense - and I am proud to say that I am now Dr Kidger!
Doing a PhD was easily one of the best decisions of my life. It's been huge fun from start to finish. (Thesis available soon!)
Big thanks to
@DavidDuvenaud
and Ben Hambly for being my examiners.
@PatrickKidger
just passed his PhD defense, which I was honored to be the external examiner for. It was a real pleasure. His thesis contributions, which included lots of open-source software, are an example of academia at its best. Congratulations, Dr. Kidger!
1
1
58
15
3
193
🌟Annnnnnouncing... Quax!
...aka "Yet Another One of My
#JAX
Libraries"!
⚡️JAX + multiple dispatch + custom array-ish types⚡️
- LoRA matrices
- named arrays
- unit systems
- symbolic zeros
- ... !
All via a custom JAX transform, "quaxify"!
GitHub:
1/2
6
23
190
Announcing the
#ICLR2023
workshop on "Physics for Machine Learning"🔥
Send us: equivariant NNs, Lie algebra approaches, neural ODEs, fluid/molecular/particle/multi-scale physics - etc!🤖
Site:
OpenReview:
Deadline 3rd Feb!⚡️
1/2
2
34
187
New paper:
"Neural CDEs for Long Time-Series via the Log-ODE Method"
GitHub:
arXiv:
Reddit:
We process very long time series of length up to 17k!
1/
1
36
186
New
#ICML2021
paper: "Neural SDEs as Infinite-Dimensional GANs"!
Showing fundamental connections between SDEs and GANs, we train Neural SDEs as incredibly flexible models for time series.
🧵: 1/n
(I'm really proud of this paper!)
1
28
173
Soliciting recommendations!
What is the current best book for a SWE moving into deep learning?📚
Ideally covering the basics (e.g. backprop, adam, train/val/test splits) and more advanced stuff (e.g. modern transformer architectures, diffusion models).
Asking for a coworker!🔥
28
7
160
I wrote a post on score-based diffusions!🌟
With a twist: the post is only a single paragraph long! (+ bookends and footnotes) There's endless complicated "explainer" articles out there. But this is the concise intuition I've been giving in person. WDYT?
6
26
161
Huge life update, round 2: 💫
I now live in the Bay Area!
San Francisco / Stanford/ Berkeley/ MV / etc. folks: send me a message and let's organise some coffee ☕️
(Still working for Google X 😎)
6
0
153
⚡️New video!⚡️I was invited to give a talk "On Neural Differential Equations" at CMU's SciML seminar.
I discuss:
- Recent work: Neural SDEs
- Recent work: Diffrax (
#JAX
software)
- Background on Neural ODEs, Neural CDEs.
- + My doctoral thesis!
WDYT?
0
21
154
The
#NeurIPS2021
workshop on Deep-Learning-meets-Differential-Equations is coming up! (14th)⚡️
For the panel discussion (on solving DEs with DL), submit questions in advance:
Note the schedule: accessible from around the world, not just US timezone!⏰️
2
35
148
Me: I should be doing math/ML research.
Also me: distracted from research making fun open-source projects.
Also also me: distracted from open-source by plotting the entire history of their GitHub stars! ⭐Which will break 1k first? Will Diffrax overtake Equinox again? etc...!⚡️
7
9
149
Here's a generative score-based diffusion model (
@YSongStanford
), trained on MNIST, written in
#JAX
.
Idk of other easy-to-use JAX implementations - perhaps a good starting point for new projects! Uses an MLP-Mixer (
@giffmana
@__kolesnikov__
) as the model
2
18
147
New paper, with
@RickyTQChen
!
"Hey, that's not an ODE": Faster ODE Adjoints with 12 Lines of Code
We roughly double the training speed of neural ODEs.
1/
1
30
142
PSA that
#JAX
now has reverse-mode autodifferentiable while loops!🎉🎉
`equinox.internal.while_loop(..., kind="checkpointed")`.
Source code + docstring:
4
6
142
A recent Reddit post on the difference between neural network libraries for
#JAX
. In which I advocate for Equinox🌓!
Reddit:
GitHub:
4
15
141
Equinox now has some beautiful documentation available at ! 🔥🔥
#JAX
neural networks have never been so easy to use.
GitHub:
1/4
2
10
130
Interested in doing a PhD on neural diffeqs / related topics?
@CristopherSalvi
is hiring a student (from maths/physics/stats/CS with PyTorch/JAX/etc. experience) for topics in diffeqs, graph NNs, causality!
Part of the Oxford-Imperial Random Systems CDT. DM him for details!
1
15
94
"Eqxvision" is a new
#JAX
library for computer vision!⚡️
Reddit:
GitHub:
Built by @|paganpasta, it's now at feature-parity with torchvision for classification models. (With segmentation, object detection etc. on the way!)
1/2
1
17
91
(Time for quite a niche post.)
A lot of folks think
#JAX
is based around pure functions. But it needn't be so!
With just a sprinkle of magic fairy dust, here's an example of a stateful function, that works even under JIT!
Source:
2
8
91
Reminder about our
#ICLR2023
workshop "Physics for ML"!
⚡️Submission deadline upcoming **3rd February**
📄4 pages long
🌍You can submit your published ICML work if it's interesting! (we're non-archival)
🕵ML-for-physics can also be snuck in
Submit here:
3
13
92
Some cool work developing Neural Partial Differential Equations!
See also
@zzznah
's reply with their elegant 6-page reaction-diffusion NDPE.
(I have no affiliation with the authors, I just like seeing how we're starting to move from NODEs/NCDEs/NSDEs on to v. practical NPDEs.)
📢Hot off the presses: we present **DiffusionNet** for simple and scalable deep learning on surfaces.
The networks generalize by construction across different samplings, resolutions, and even representations. Spatial support is automatically optimized as a parameter! 🧵👇 (1/N)
6
140
832
1
6
91
A reminder!⭐️The deadline for the
#NeurIPS2022
workshop
"Symbiosis of Deep Learning and Differential Equations"
...is coming up soon! 24 September.😃
Website: Accepting submissions on neural ODEs, score-based diffusions, learnt numerical methods etc!
1
20
90
Hard to express *just how cool* meeting all of you through academic twitter has been. (I work at Google X, and live on another continent, due to this site!!)
So if this is the end: ty, it's been a blast🔥
Come find me on🦣
and/or in real life at
#NeurIPS
!
8
4
87
The Lineax paper is now on arXiv!⚡
Fast, extensible, linear solves and least squares in
#JAX
.
It'll be appearing at the NeurIPS AI4Science workshop. (Come along and say hi if you see us there!)
And congratulations to Jason on their first paper! 😀
⭐ Lineax is now on arXiv! ⭐
If you’re doing linear solves or linear least-squares in JAX, give it a shot today!
Lineax
is fast ⚡️,
has new solvers (eg. QR, tridiagonal),
supports general linear Operators.
github:
arXiv:
1/n
2
13
116
1
3
86
Announcing Equinox v0.1.0! Lots of new goodies for your neural networks in JAX.
-The big one: models using native jax.jit and jax.grad!
-filter, partition, combine, to manipulate PyTrees
-new filter functions
-much-improved documentation
-PyPI availability!
A thread:
1/n 🧵
3
9
80
Here's a blog post!❄️
"How to handle a hands-off supervisor"
I was pretty independent in my PhD. This is what I found helped the most. :)
Discussing managing time, self-promotion, finding mentors, and above all: when stuck, how to unstick yourself!🌟
3
6
80
Equinox now has some new examples🥳:
- Intro: CNN-on-MNIST (suitable for those new to JAX!)
- Advanced: a U-Net impl.
- Advanced: BERT!
All provided by community members. Thanks to
@ArturAGalstyan
@MahmoudAsem
@jj_rader
for contributing to this release!
3
9
74
For our
#NeurIPS2021
paper, I'm not recording a SlivesLive talk. I never watch these; no-one else I know does either.
It seems far easier + more efficient to promote my work here on Twitter instead.
(On that note: 😅)
Agree/disagree? Anyone else do this?
16
4
72
⭐️The newest Equinox release includes something very special: runtime errors for
#JAX
!⭐️
And if you set the `EQX_ON_ERROR=breakpoint` environment variable, they will automatically open a debugger on error.
Docs:
GitHub:
1/
1
15
73
⚡This is pretty awesome.
Announcing torch2jax! Run PyTorch code natively in JAX. 🤝
Mix-and-match PyTorch and JAX code with seamless, end-to-end autodiff, use JAX classics like jit, grad, and vmap on PyTorch code, and run PyTorch models on TPUs.
14
74
563
0
3
74
A lot of replies to this are folks disagreeing; saying that software skills aren't most important.
I disagree with those disagreeing! 100% the top advice I give is to be good at software dev. Unless you do pure math then it's what you spend most time doing - so be good at it!
This is probably well-known in some circles but not everywhere.
The most important skill for Research Scientists in AI (at least at
@OpenAI
) is software engineering.
Background in ML research is sometimes useful, but you can usually get away with a few landmark paper.
38
194
2K
7
1
71
Quick advert for anyone in my audience doing advanced
#JAX
stuff!🌟
Version 0.9 of Equinox has just been released!
- Runtime errors;
- XLA sub-graphs (=faster compile times via no-inlining);
- New filtered interface on to custom primitives.
- more!
2
2
69
Alright y'all I am up at very-early-o'clock to get myself to
#neurips
#neurips2022
!
Also interested in:
🧪scientific ML?
🧮differential equations?
🤖JAX, Julia?
🖥open-source software?
🧬computational biology?
...send me a DM and let's meet up!
1
0
65
🔥Hello everyone - I am in BOSTON!⛵️☕️
Who's around and wants to chat sciML/proteins/startups/open-source? DM me!
Those of you at the PEGS conference, come along to our cofounder Eli's talk on ML-for-protein-design at Cradle later today! 🤖+🧪:
3
2
68
Excited to announce `torchcde`!
Built on torchdiffeq, this is a library for solving controlled differential equations.
Particularly useful for _neural_ CDEs, which are like RNNs but better! :D
1/
1
18
64
@alishbaimran_
If by "fundamentals" you mean a first intro, then I'd point out
(using Equinox as the neural network library.)
And for a more-nontrivial example -- here this a score-based diffusion -- then
If by "fundamentals" you mean JAX's own
2
8
66
Here's a fun picture I made today!
A continuous normalising flow () continuously deforms one distribution into another distribution.
The lines show how particles from the base distribution are perturbed until they approximate the target distribution.
4
5
64
And of course, all accompanying code is provided -- available as the examples in the brand-new Diffrax software library! Your one-stop-shop for numerical differential equation solvers in
#JAX
.
GitHub:
Documentation:
13/n
2
5
65
Quick advert: the latest Equinox releases have added:
⭐️Stateful operations
➡️Docs:
➡️Example (autoregressive attention):
⭐️Runtime errors
➡️`equinox.internal.error_if`
to
#JAX
! 🎉🎉
GitHub:
0
7
61
Reminder that 'The Symbiosis of Deep Learning and Differential Equations'
#NeurIPS2021
workshop deadline isn't too far away!
Deadline: September 17th [10 days away]
Website:
OpenReview:
(w/ track for advertising published papers!)
1
15
53
This is really cool! A new
#JAX
library for probabilistic modelling (in this case over structured distributions.)
Right now I think the docs are a bit sparse, but for the curious I recommend taking a look at the paper -- this does a pretty good job selling it to me.
Today we are open-sourcing SynJax which is a JAX library for efficient probabilistic modeling of structured objects (sequences, segmentations, alignments, trees...). It can compute everything you would expect from a probability distribution: argmax, samples, marginals, entropy...
1
65
280
0
2
54
⚡️ New paper! ⚡️
"Equinox: neural networks in JAX via callable PyTrees and filtered transformations"
Accepted
@diffprogramming
#NeurIPS2021
!
Elegant and simple neural networks in
#JAX
.
Paper:
Library:
1/n🧵
3
10
50
When you're trying to make a point in your documentation.
()
3
6
47
Me: ML "scientist", have spent the past week trying to figure out why my linear model won't train🤡
My astrophysics scientist friends: simultaneously release 10 papers simulating supermassive black holes, microquasar jets, gamma-ray binaries.🕳️✨☄️
:D
Thrilled to share our latest work, where together with
@jaj_garcia
, Dom Walton,
@RicardaBeckmann
and the
@HEXP_Future
Team we demonstrate HEX-P's unique ability to study SMBH growth histories with X-ray reflection spectroscopy ✨
1
2
20
0
0
45
🌟I am living in Palo Alto for the next few weeks!
Stanford folks, send me a DM and let's get coffee! I'm particularly interested in anything in sciML 🧪/ open source 🤖/ biology.🦠️
(Cute photo of Peanut the dog to get your attention: )
1
2
44
This is a really cool example of an interesting question in optimisation -- and one which
@packquickly
used Optimistix (GitHub: ) to investigate.
Have a read. :)
Tikhnov regularised trust-region methods (*cough* Levenberg-Marquardt) oddly use two different approximations to the objective function at each step.
One regularised, one not.
What if we just regularised both?
1/
1
6
35
0
4
42
I've written the content in Markdown, and then use `markdown` and `pymdownx` to parse that into HTML. The styling is based on the SCSS files from `mkdocs-material`.
And now e.g. I can change to landscape and things reflow automatically. Literally no changes to content or code.
3
4
42
Here's a fun new feature in Diffrax.⚡️
Automatically-generated BibTeX references for the differential equation solvers you use!
Just substitute your usual `diffrax.diffeqsolve` for a `diffrax.citation` instead.
GitHub:
Docs:
#JAX
0
3
41
And the workshop website is now ready! Includes CfP, schedule, invited speakers etc.
(We're also hoping to use the workshop as a venue to promote already-published papers strongly within the workshop's scope; check that out too.)
Our 'The Symbiosis of Deep Learning and Differential Equations' workshop has been accepted for
#NeurIPS2021
!
Send us your work on data-driven dynamical systems, neural differential equations, solving PDEs with deep learning etc.
Tentative submission deadline Sept. 17.
4
50
307
2
7
42
I am now also on
#mastadon
🦣! Like the rest of you, I'm jumping on the bandwagon to hedge my bets against Twitter becoming nonprofitable+collapsing.
>Me:
I've left a 1-toot guide for anyone else looking to get started.
#academicchatter
#academictwitter
3
3
40
I've just had a look through this, and I have to say that I'm really impressed!
A framework for training large models, top-notch software engineering, and best of all -- built using Equinox! :)
Today, I’m excited to announce the release of Levanter 1.0, our new JAX-based framework for training foundation models, which we’ve been working on
@StanfordCRFM
. Levanter is designed to be legible, scalable and reproducible.
6
91
412
1
1
41
Reading old computer science papers is a lot of fun.
"30 megabytes of storage ... is clearly debilitating on larger problems"
2
0
40
@tw_killian
😀 Happy to hear that you like it!
ODE - base case. A continuous time version of a ResNet.
CDE - when you add in a time-varying input. A continuous time version of an RNN.
SDE - when you want a generative model; think of this like a GAN. Noise goes in, sample comes out.
1
6
39
Okay, let's wrap this up. If you're studying NDEs and want a reference text, then maybe this is it?
231 pages of everything you ever wanted to know about N ordinary DEs, N controlled DEs, N stochastic DEs, and N rough DEs.
Once again, link here:
18/18
2
0
37