Miles Cranmer
@MilesCranmer
Followers
13K
Following
5K
Media
570
Statuses
3K
Assistant Prof @Cambridge_Uni, works on AI for the physical sciences. Previously: Flatiron, DeepMind, Princeton, McGill
Joined September 2011
I'm super excited to share a new initiative I am a part of!. Announcing: Polymathic AI 🎉. We are developing foundation models for scientific *data*, such that they can leverage shared concepts across disciplines. 1/6
16
215
1K
John von Neumann: "with four parameters I can fit an elephant". Meanwhile, this paper "How to fit any dataset with a single parameter" . Here's a function with a *single* parameter. Even worse: it's differentiable and continuous!
41
601
2K
Very excited to share our new paper "Discovering Symbolic Models from Deep Learning with Inductive Biases"!. We describe an approach to convert a deep model into an equivalent symbolic equation. Blog/code: Paper: Thread👇.1/n
28
519
2K
Three years ago, I started working on an easy-to-use tool for interpretable machine learning in science. I wanted it to do for symbolic regression what Theano did for deep learning. Today, I am beyond excited to share with you the paper describing it!. 1.
38
371
2K
Here's a condensed version of the matplotlib cheatsheets so it can fit a desktop background.(. Full image: and vectorized .svg, with the non-standard fonts outlined: Thanks @NPRougier et al for making it!
7
381
2K
It's crazy how over time I have slowly replaced all of my command line tools with Rust equivalents 🦀. - cat → bat.- pip → uv.- grep → ripgrep.- htop → zenith.- fswatch → watchexec. Any other good ones?.
82
114
2K
Could machine learning rediscover the law of gravitation simply by observing our solar system?. With our new approach, the answer is *YES*. Led by: @PabloLemosP .With: @Niall_Jeffrey @cosmo_shirley @PeterWBattaglia.Paper: Blog:
23
308
1K
If you’ve never tried it, is the single best explanatory tool for neural networks. An essential demo for any deep learning course!. I still notice improvements in my intuition just by tinkering with it. From @dsmilkov @shancarter.
9
262
1K
Life update: this fall I will be joining the University of Cambridge as Assistant Professor!. I will be appointed as joint faculty between DAMTP and the Institute of Astronomy 🚀
61
20
1K
Today I learned you can write numbers like this in Python (!!). Makes it easier to read long numbers by separating digits into groups, just like 1,000,000. It’s so esoteric that Google Colab doesn’t even color it correctly!
30
120
1K
A matplotlib trick that I wish I learned a long time ago:. To adjust resolution of figures, rather than using. plt.figure(figsize=(8, 8)). followed by a tweaking of every font size, you can just increase the resolution with:. plt.figure(dpi=300).
21
133
1K
Amazing. VSCode LaTeX Workshop has dark mode for *PDFs*! It even inverts the figures!.
9
130
1K
I'm starting a curated list of interactive machine learning demos:. Looking for more suggestions!. My plan is to incorporate some into the ML modules of Cambridge's new MPhil in Data Intensive Science, as a way to hone students' intuition.
19
219
1K
Why 'I don’t know' is the true test for AGI—it’s a strictly harder problem than text generation!. This magnificent 62-page paper ( formally proves AGI hallucinations are inevitable, with 50 pages (!!) of supplementary proofs.
45
141
948
I am blown away by Such a useful tool for research. This in-browser graphical LaTeX tool gives you free-form drawing (tikz export), WYSIWYG rendering, symbol shortcuts, and even picture-based symbol search. I might even write full papers in this.
7
126
857
TabNine is awesome: It suggests code completions in real-time using deep learning conditioned on your existing code. Free plugins for Jupyter, vim, emacs, sublime, and VS. Really enjoying it so far. Thanks @ykilcher for pointing it out!
17
177
837
My favorite way to explain a normalizing flow:. - There's a crowd of people; each is a sample of the data distribution. - Everybody takes a step in some direction according to a neural net.- In steps, the net tries to direct the crowd to form a Gaussian without bumping each other
12
79
702
If you use PyTorch, I highly recommend checking out @huggingface's Accelerate: It's as minimal as it is powerful: multi-GPU/TPU training, while still preserving your original training loop!. you can even run multi-device from a Jupyter notebook:
4
117
658
The more I use Julia, the more Python and its numeric libraries look like a Victorian-era stagecoach with jet engines duct-taped to it, each pointing a different direction (=mutually incompatible). It's such a weird ecosystem, and makes it so much harder for users to contribute.
31
58
594
Just learned about Python Fire, and wish I had heard about it years ago. Seems like an amazing library for productivity!. Fire turns any Python object—function, class, etc—into a command line interface:. Gone are the days of argparse.ArgumentParser and sys.argv.
12
103
575
I just generated an entire slide deck using ChatGPT, by giving it a list of slide titles. Productivity 📈
16
69
546
Wow, @sagemath's LaTeX package is amazing. It bridges the gap between symbolic math software and LaTeX presentation. Uses SymPy as the algebra backend and formats the output into the pdf. Wish I knew about this in undergrad!
12
105
521
This is a nice package for making pyplot animations more intuitive: All you do is call "camera.snap()" every time you re-do the plot.
4
127
512
Oh. My. God. Can someone please port this to PyTorch?.
Excited to share Penzai, a JAX research toolkit from @GoogleDeepMind for building, editing, and visualizing neural networks! Penzai makes it easy to see model internals and lets you inject custom logic anywhere. Check it out on GitHub:
5
61
518
I think the best part of today's news is it will encourage more AI hires in physics departments.
BREAKING NEWS.The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”
32
47
503
Wow, JAX is amazing. Thanks for introducing me @shoyer. It's essentially numpy on steroids: parallel functions, GPU support, autodiff, JIT compilation, deep learning. #NeurIPS2019 .
6
97
477
Our paper demonstrating the power of Bayesian Neural Networks for planetary dynamics comes out in PNAS today!. (open access). This paper explores a match made in heaven: chaotic systems and Bayesian neural networks. Thread:
4
97
479
Just-in-time compiled languages are not supposed to be this fast at startup. The speed of the upcoming Julia update is ridiculous. (Julia 1.10-alpha vs. Python 3.11)
18
52
484
Awesome interactive demos of different MCMC algorithms:.
2
118
450
Happy to announce SymbolicRegression.jl, a Julia package for learning equations via evolution! It supports distributed computing, allows user-defined operators (even discontinuous!), and exports to SymbolicUtils.jl. v0.4+ of PySR uses this as backend.
6
82
411
So 1) Lagrangian/Hamiltonian NNs enforce time symmetry, 2) Graph Nets enforce translational symmetry, and 3) Group-CNNs enforce rotational symmetry. But are there any NNs that can enforce an arbitrary learned symmetry?.@wellingmax @DaniloJRezende @KyleCranmer?.
20
72
409
Matplotlib palettes are useful for science, because they show contrast well, but I find them not to be visually pleasing. Are there any good libraries with aesthetics-focused matplotlib colorschemes?.
40
20
393
Very excited to present our new work: we adapt Bayesian neural networks to predict the dissolution of compact planetary systems, a variant of the three-body problem!. Blogpost/code: Paper: API: Thread: 👇
5
61
364
The forced hash collision idea from InstantNGP ( remains one of the most creative ideas I've ever seen in deep learning. I tried to explain it to someone today and had no idea where to start. it's too unconventional (in a good way!). And it works well!
4
52
379
This paper distills neural networks onto FPGAs with symbolic regression, obtaining a 5 NANOSECOND inference time!!. Super cool application of PySR and awesome work by the lead authors 🙌
2
70
366
I love how @andrewwhite01 just casually released the greatest academic search tool ever created:. Literature reviews on steroids?.
I packed-up a full-text paper scraper, vector database, and LLM into a CLI to answer questions from only highly-cited peer-reviewed papers. Feels unreal to be able instantly get answers by an LLM "reading" dozens of papers. 1/2
4
37
360
Here's a thread on lesser-known tools and packages that I could not live without, starting with Python. (suggestions are very welcome!). einops:.- - Easily-interpretable reshapes + tiling + aggregations for numpy/torch/tf/etc. 1/n
3
55
354
Master's program opportunity – Apply to be a @GoogleDeepMind Scholar at Cambridge in our DIS program!. I see this as an ideal program to launch a career in AI for the physical sciences, so am very excited that DeepMind have offered support for underrepresented students 🎉. More
8
60
360
1/10 Very excited to present Lagrangian Neural Networks, a new type of architecture that conserves energy in a learned simulator without requiring canonical coordinates. w/ @samgreydanus, @shoyer, @PeterWBattaglia, @DavidSpergel, @cosmo_shirley:
11
92
358
I regret not reading through the full LaTeX physics package earlier; so many more features than I realized. Many commands that I usually define by hand. e.g., some macros for partial derivatives:.
2
59
349
This jax function is pretty cool. Higher-order derivatives without repeated autodiff:
3
45
342
Job alert! 🚨. We are building a *Foundation Model for Science*. @SimonsFdn + @FlatironCCA are supporting PhD internships + faculty sabbaticals!. w/ @cosmo_shirley @kchonyc @Tim_Dettmers @oharub ++. Interested in building "ScienceGPT" with us? Please apply! (links in 2nd tweet)
12
76
346
I'm late, but weight averaging seems like a great trick for improving DL generalization (@Pavel_Izmailov et al). Take a pretrained model, do SGD about minima, and average weights. Thanks @andrewgwils for recommendation!. Found a big improvement in my tuned model at zero cost:
3
52
338
🧵 Could this be the ImageNet moment for scientific AI?. Today with @PolymathicAI and others we're releasing two massive datasets that span dozens of fields - from bacterial growth to supernova!. We want this to enable multi-disciplinary foundation model research.
12
88
336
Okay, Pluto.jl is the best part of Julia I've seen so far. Absolutely game-changing. It's Jupyter, but reactive: change a variable, and the entire notebook updates. This means you can do things like use a slider to vary a parameter in some cell. and see all your plots change!
8
48
328
My lectures this week include 'Best practices' and I will be assigning @karpathy's neural net training blog for reading material :). Really an *essential* read for every practitioner!
Very excited to start teaching my deep learning course at Cambridge this week, as part of our Data Intensive Science MPhil!. Teaching the first part from @SimonPrinceAI's "Understanding Deep Learning" book, which has quickly become one of my favorite textbooks in *any* field.
1
35
316
I'm really starting to like @michael_nielsen's strategy of reading papers. Write down a question about the background or results, find the answer, distill, repeat. It feels like test-driven development. Write a test, make it work, refactor, repeat.
7
35
320
ChatGPT has almost completely replaced StackOverflow for me at this point. Getting context-specific answers with detailed explanations that I can iterate on in a pair programming-like fashion is incredible. The crazy part is this is only GPT-3.5. .
15
33
311
In a neural network, is there a type of regularization which encourages one learned feature to be independent, **including nonlinearly,** of other features in the same layer?. I can’t use a bottleneck or sparsity constraint—I actually want to maximize the dimensionality!.
44
37
315
Using paperqa, I fed GPT every paper in my Zotero library and asked: "What are some ways machine learning can be used in observational astronomy?". It generated the entire literature review below. Not bad at all!. with @andrewwhite01's
13
45
313
This is a really nice review and independent evaluation of the many available neural net optimizers: It’s quite extensive!
4
69
304
Happy to announce that our work on converting deep models to symbolic equations has been accepted to NeurIPS! 🍾. @PeterWBattaglia @cosmo_shirley @DavidSpergel @KyleCranmer.
Very excited to share our new paper "Discovering Symbolic Models from Deep Learning with Inductive Biases"!. We describe an approach to convert a deep model into an equivalent symbolic equation. Blog/code: Paper: Thread👇.1/n
3
41
294
I made a tutorial on simulation-based/likelihood-free inference for scientists using PyTorch-based "sbi"!.+ colab notebook. Thanks for putting together this awesome set of libraries .@jakhmack, @deismic_, @janmatthis, @conormdurkan, @driainmurray, et al.
5
73
274
After two months of studying, I have just passed my comprehensive exam at Princeton 🎉 Officially a PhD candidate!. Excited to get back to doing research!.
16
2
266
Why do ML people still guess equations for scaling laws when symbolic regression exists???.
12
19
273
Made a functional SymPy->JAX converter equivalent :). Works with grad, vmap, jit, etc. PySR/SymbolicRegression.jl can automatically convert discovered expressions to vectorized JAX models now; will add PyTorch soon.
Put together a micro-library for turning SymPy expressions into PyTorch Modules. Symbols becomes inputs, and floats become trainable parameters. Train your SymPy expressions by gradient descent!
2
47
270
Are there any review articles which study the importance of open-source software for the sciences?. Relatedly, here's a great quote from Freeman Dyson, which I think also underlines the importance of free and open-source software.
7
44
235
PyTorch Lightning’s greatest strength is that it implements a vast amount of deep learning tips and tricks which would typically take years to pick up. e.g., previously I'd never heard of gradient clipping. I turned it on and my model's NaNs vanished!
I have been playing around with @PyTorchLightnin and I am pleasantly surprised!. Very good level of abstraction if you want full control over the model & some production-level tools, eg, many loggers and quick debug iterations. Kudos to the team! Looking forward to 1.0. 💪.
5
34
238
Happy to share I will be doing a research internship at @DeepMind from July-November with @PeterWBattaglia and @DaniloJRezende. Excited to work on some new approaches to AI for Physics!.
8
5
237
Very cool paper from @EmtiyazKhan:. Relatedly, here's a great blog post that helped me with intuition about natural gradients:
Our new paper on "The Bayesian Learning Rule" is now on arXiv, where we provide a common learning-principle behind a variety of learning algorithms (optimization, deep learning, and graphical models). Guess what, the principle is Bayesian. A very long🧵
6
45
234
The more I use XLA and JAX, the more I see the true potential of its python API: you can do all the crazy pure-python meta-programming you want, so long as the moving parts depend on static arguments, and the optimizer boils it down to the actual tensor operations. So nice!.
5
19
227
Want to have an existential crisis? Here you go:. There's a theory that humans aren't actually conscious. We've just evolved to think we are because it boosts self-preservation. In other words, the "feeling" of consciousness is just from natural selection, but isn't real.
65
12
231
Regarding Dalle and Imagen:. These systems are *amazing*. However,. I (selfishly) wish that all of that ML expertise and compute was focused on solving scientific problems, rather than generating panda art!. Yes, it advances the field, but why not solve science simultaneously?
19
21
214
SymbolicRegression.jl → 1.0 🎉 . After several years of work, I'm thrilled to announce some major new features! Let me show you what's possible now:
5
43
219
Giving mock general exams today at Princeton Astro (oral), and reviewing my favorite tricks:. km/s ≈ pc/Myr.year ≈ 10^7.5 seconds.1" ≈ 5 μrad.R_earth ≈ R_jup/10 ≈ R_sun/100.G ≈ 40 AU^3/(Msun year^2).m_e ≈ 0.5 MeV/c^2 ≈ m_p/2000.1200 nm => 1 eV. What are other good ones?.
31
31
197
Required reading for anybody using PINNs:. I think PINNs are an exciting idea but many use cases are perhaps better suited to learned NN prediction (for unresolved scales), or just standard numerical integrators (resolved scales). (1/2)
4
18
206
Interested in doing a PhD on AI for the physical sciences at Cambridge? I am taking PhD students for 2024!! Please find information below, including a list of projects:. (Deadline typically early December or January, depending on program)
1
47
204
My Simons Presidential Lecture is up on YouTube!. In this talk I make the argument that 'The Next Great Scientific Theory is Hiding Inside a Neural Network'.
4
32
197
Apparently you can create a global .gitignore!?!. I've been making local ones like a fool this whole time.
4
19
194
Giving the Presidential Lecture tomorrow at @SimonsFdn @FlatironInst:. "The Next Great Scientific Theory is Hiding Inside a Neural Network" Will be in NYC until the 10th – please get in touch if you would like to chat!
8
29
188
I love this paper as an example for how neural networks can cheat, and encode more information into a single neuron than you would expect.
2
13
182
Happy to share that the "Multiple Physics Pretraining" paper won the Best Paper Award at the AI for Science NeurIPS workshop! Congratulations to @mikemccabe210, @liamhparker, @BrunoRegaldo @oharub for leading the effort, and everybody in the @PolymathicAI team!
Honored to receive best paper for MPP at the @AI_for_Science @NeurIPSConf workshop with my teammates @PolymathicAI! Thanks to everyone who stopped by our poster for the great discussions and to the organizers for running such an interesting workshop! #AI4Science #NeurIPS2023.
2
21
183
ML-accelerated scientific discovery in action!. This new paper in ApJ Letters uses PySR to discover a new relation between supermassive black hole mass and properties of its host spiral galaxy:. Extremely cool work!!.
0
32
182
Words cannot express the perfection of @TuringLang for probabilistic inference. It's somehow both intuitive and concise without sacrificing any expressiveness. (Also blazingly fast, of course). Doing my first real project with it and having a blast.
4
19
183
1/10 This was a phenomenal discussion. I have many more questions than answers now but I think that's a good thing. Here's a list of some interesting papers mentioned.
So 1) Lagrangian/Hamiltonian NNs enforce time symmetry, 2) Graph Nets enforce translational symmetry, and 3) Group-CNNs enforce rotational symmetry. But are there any NNs that can enforce an arbitrary learned symmetry?.@wellingmax @DaniloJRezende @KyleCranmer?.
2
44
171
Essential Overleaf trick: you can have Overleaf run *arbitrary* code before each compilation of the PDF!. Write the following code into a file called .latexmkrc in your project, replacing "custom_command" with whatever (e.g., latexdiff).
6
13
167
ChatGPT just fixed one of my matplotlib scripts (colorbar was too big). Mind blown. It really feels like the evolution of the search engine.
4
9
166
Is there a way to constrain a neural network to be a harmonic function? (zero Laplacian).
23
31
163
Excited to attend JuliaCon for the first time this year!.Will be giving a talk on SymbolicRegression.jl: + uses in science. This will be the first SR talk where I dive into low-level engineering details. Looking forward to learning from other attendees!
3
17
164
Are you a PhD student who is (1) interested in working on foundation models for science, and (2) experienced with deep learning software?. There is a 1-year internship at Flatiron Institute (NYC) to work on @PolymathicAI!. (deadline: Nov 30!)
2
35
167
Very excited to start teaching my deep learning course at Cambridge this week, as part of our Data Intensive Science MPhil!. Teaching the first part from @SimonPrinceAI's "Understanding Deep Learning" book, which has quickly become one of my favorite textbooks in *any* field.
9
14
164
I am entering the faculty job market for 2023! Very eager to find a position at the intersection of astro/physics and machine learning/data science. If you happen to see something relevant, please forward to mcranmer@princeton.edu - thanks!.
5
9
160
Deep learning research seems to suffer from periods of frenzied activity on niche topics. I think social media worsens the collapse into targeted research problems because it makes FOMO so much stronger. But long-term it is terrible for creativity in the field. (1/3).
7
11
163
1/2 Why isn't it more common to do explicit Hamiltonian MCMC on a Bayesian Neural Network's weights, with eg the initial condition = the loss minima found via SGD? I'm playing around with one in JAX and it seems to be working reasonably even with 5 chains:
7
20
158
Tullio.jl makes pretty much any tensor operation a one-liner. It's like a grown-up version of einsum!.
4
21
156
BayesNet seems like a really nice LaTeX package for drawing clean probabilistic graphical models with minimal effort. Wish I heard about it earlier!.
2
30
154
PyTorch-style deep learning in Julia!. As a longterm PyTorch user I am really happy to see this is possible in @FluxML. The key advantage is that Julia *itself* is autodiff-ready, so you can compute gradients through a complex library without needing a rewrite in a DL framework.
3
17
155
Happy to announce differential operators for PySR + SymbolicRegression.jl!. This means you can literally just. evolve an integral:
4
20
152
Wish I found this a while ago:. mamba is a much faster backend to conda, with an identical set of commands, same package servers, etc. My 30-min environment build is now <1 min with zero changes to the yml file. .
2
28
153
It's amazing how Enzyme is this much faster than JAX for even simple operations!. (Am I doing something wrong, or is differentiating through optimized assembly code really that much faster??)
The idea behind Enzyme differentiation is so cool. It literally performs autodiff through optimized assembly code*, which gives faster derivatives!. Q: Would this let you differentiate in-place array operations?. *(LLVM IR, not machine code).
10
17
154
Just discovered this 9-digit approximation to π with PySR's genetic algorithm. is this a known formula?
11
10
148
Okay, here is a function for doing this (modulo shading) in LaTeX, without external illustration tools: This is what .$$\labmat{2}{3}{X} \cdot \exp(\labmat{3}{2}{Y})$$.looks like:. Thanks @AgolEric @rmpnegrinho for pointers!
That's a nice way of writing equations (I sometimes do this in lectures). From ICLR 2021 submission ("An attention free transformer"),
2
30
149
ChatGPT is trained on ~500 GB of text. ~1 byte per character = 5e11 characters.~2000 characters per page = 2.5e8 pages.~0.1 mm thickness per page = 25,000 meters. So ChatGPT is trained on a book that is 25 km high. (more than double the cruising altitude of commercial planes).
4
32
146
PySR paper is coming out tonight. I'm wondering. should I do a science-themed announcement today, given that ML people are at ICLR, (and then an ML-themed announcement next week)?
2
16
145
New PySR release! The new Python↔Julia interface is massively improved thanks to PythonCall.jl. Julia can now be used seamlessly as a general backend for writing fast Python libraries!
3
22
141
Happy to share our paper on AI for observational astronomy via our new resource allocation algorithm!. "Unsupervised Resource Allocation with Graph Neural Networks". Blog/code: Paper: w/ @peter_melchior @iamstarnord . Thread 👇.1/n
5
24
143