Stephan Hoyer
@shoyer
Followers
7,836
Following
755
Media
63
Statuses
2,960
Senior staff engineer at Google Research building AI weather/climate models. Creator of @xarray_dev and scientific Python contributor. He/him.
San Francisco, CA
Joined August 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Bronx
• 963114 Tweets
Rafah
• 401715 Tweets
Xavi
• 302827 Tweets
Flick
• 158181 Tweets
Travis
• 123478 Tweets
LEAVE SEVENTEEN ALONE
• 103293 Tweets
Ten Hag
• 98465 Tweets
Laporta
• 91144 Tweets
Memorial Day
• 63278 Tweets
#SRHvsRR
• 62154 Tweets
Congratulations LISA
• 49904 Tweets
العدل الدوليه
• 46597 Tweets
#تتويج_الهلال
• 36072 Tweets
Coutinho
• 32920 Tweets
QSMP
• 32092 Tweets
Sokak Köpekleri Toplatılsın
• 28202 Tweets
Super Size Me
• 27215 Tweets
Morgan Spurlock
• 27061 Tweets
INEOS
• 23764 Tweets
Neto
• 21059 Tweets
Ergin Ataman
• 20340 Tweets
Yunan
• 17004 Tweets
Mourão
• 15710 Tweets
Tyga
• 15205 Tweets
Yarın AdanaMimarSinandayız
• 14269 Tweets
Abhishek Sharma
• 13291 Tweets
Gove
• 12986 Tweets
الدوري الاقوي
• 12826 Tweets
Fenerbahçe - Panathinaikos
• 12410 Tweets
فهد بن
• 12113 Tweets
Kylie
• 11801 Tweets
Pinned Tweet
New open source release from my team at Google: Dinosaur, a differentiable dynamical core for global atmospheric modeling, written in JAX:
Dinosaur is a core component of NeuralGCM and we hope it is useful for the weather/climate research community.
7
66
439
Are you a PhD student interested in machine learning and numerical modeling for weather & climate?
My team at Google Research is looking to hire a student researcher for this summer (and likely beyond) in either Mountain View, CA or Cambridge, MA.
18
252
866
I'm really excited to share this project that my team at
@GoogleAI
has been working on for the past year.
We show that ML + TPUs can accelerate fluid simulations by up to two orders of magnitude without compromising accuracy or generalization.
1/2
Excited to share "Machine learning accelerated computational fluid dynamics"
We use ML inside a CFD simulator to advance the accuracy/speed Pareto frontier
with/
Jamie A. Smith
Ayya Alieva
Qing Wang
Michael P. Brenner
@shoyer
12
123
532
16
143
822
Just hit quadruple digits for new unnamed Colab notebooks! 🎉
10
18
818
Academics often ask me if we have "academic freedom" in industrial research.
Of course we don't, but generally that isn't the real concern -- they want to know if I have control over my own destiny.
I tell them this is my favorite part about industry. 🧵
15
56
434
I love when ML researchers get excited about science, but seriously the reviewing process for scientific applications at ML conferences (e.g., ICLR) is entirely broken.
Papers with glaring errors are sailing through, without a single review from somebody with domain expertise.
20
26
353
please tell me I'm not the only one who thinks "consciousness" is not even a scientific concept
63
8
331
Gradient checkpointing (aka rematerialization) is an easy trick that can save massive amounts of memory for calculating gradients.
If you differentiate through computation involving long iterative processes (like ODE solving), learn it and make it part of your toolkit!
👇🧵
5
42
329
We're developing a new protocol (__array_function__) that allows for alternate implementations of NumPy functions (e.g., for GPU, autodiff or units support):
It's now implemented on the master branch of NumPy -- please give it a try and report back!
3
91
325
Can machine learning improve physics constrained optimization tasks, like those at the heart of numerical weather forecasting?
Happy to share some our work on "Variational Data Assimilation with a Learned Inverse Observation Operator", tomorrow/today at ICML.
🧵
6
48
305
JAX now supports Google Cloud TPUs!
I contributed this example, solving a 2D wave equation with a spatially partitioned grid. The code is remarkably simple and all in pure Python!
2
54
296
Does anyone know a good intuitive explanation for the central limit theorem? I realized the other day that even though I use it all the time I can't really justify *why* it's true.
55
25
258
My team at Google is looking to hire a PhD student intern for research on AI-based coupled Earth system modeling. This would be a full-time ~3 month position in summer or fall 2024 working in-person in Cambridge, MA with
@dkochkov1
and
@janniyuval
.
6
58
254
Something that I think is under-appreciated in the current AI mania is that more compute does not always result in better models. Sometimes, even with perfect knowledge, you can hit a wall.
A good example of this is weather prediction.
13
14
253
One lesson for writing NumPy/JAX code that took me a surprisingly long time to learn is to preserve array dimensionality whenever possible (i.e., always use keepdims=True).
3
14
237
The world's best weather forecast model is switching its numerics from double to single precision:
They reinvested the 40% runtime savings in an increase in vertical resolution from 91 -> 137 levels 😍
New
@ECMWF
forecast system still on track for 11th May
#IFS47r2
#newfcsystem
@ECMWF
Update your software to be ready!
👇
1
10
45
3
22
190
Honest question: why do ML researchers publish papers filled with giant tables?
Yes, it's nice to have as a reference for comparing to prior results, but couldn't we put nice plots in the paper and the uninterpretable raw numbers in an appendix?
22
6
180
I'm happy to share a new paper, with
@jaschasd
and
@samgreydanus
: "Neural reparameterization improves structural optimization"
We use neural nets to parameterize inputs of a finite elements method, and differentiable through the whole thing:
2
38
170
We are hiring PhD interns for my team at Google Research:
We use computational methods (especially ML) to advance research in a variety of scientific fields. For full consideration, please apply by January 15:
8
40
167
This paper "Optimal control of PDEs using physics informed neural networks" looks really nice:
Finally, an assessment of PINNs that includes a runtime comparison to classical adjoint methods!
4
20
151
I'm happy to share a new project on using machine learning for computational fluid dynamics, led by
@gideoknite
:
"Learning to correct spectral methods for simulating turbulent flows"
2
24
144
In this project led by
@leeley18
, we show that end-to-end training with differentiable physics results in extremely effective hybrid physics/ML models for density functional theory.
We discover that the prior knowledge embedded in the physics computation itself acts as an implicit regularization that greatly improves generalization of machine learning models for physics.
Please check out our recent paper:
7
114
392
2
30
145
I'm pleased to announce xarray v0.11:
This release includes:
- file-storage refactor for performance with
@dask_dev
- better support for calendars used in climate science
- lots of other miscellaneous API clean-ups, bug-fixes and enhancements
3
47
131
This year, Google's Research Scholar program for early-career professors is specifically solicitating proposals on large-language and multi-modal machine learning models for science:
Applications will open next week and are due by the end of November.
1
36
130
My amazing colleagues on the Google AI for Weather & Climate team are hiring for a research/software role:
1
25
129
It is hard for me to imagine a better feeling than passing off a project to a total stranger from half way around the world, who builds on it and takes it to new heights you never imagined 🥰
This is the true magic of open source software.
5
2
130
I hope I never read another paper claiming faster simulation with ML that only compares to sims used for training data.
Congrats, your method is faster and less accurate. So what? That costly reference simulation could almost certainly be made faster and less accurate, too.
12
13
127
I am so ready for LLMs to blow over so I can use data centers worth of TPUs for climate modeling 😂
We are in a midst of perhaps the greatest computing power buildup ever. Sure, the vast majority of it is going towards the AI products, but at some point, between two LLM trainings, someone will decide to use at least a fraction of it for other purposes. Hey, let’s simulate
55
63
808
4
6
124
In industry, you do not have complete freedom within a job, but you have freedom to pick an organization to work for that aligns with your values.
Within a (good) job, nobody wants to tell you what to do. You succeed by showing that can deliver on the organization's mission.
5
7
116
Today's PSA: don't use continuous colormaps if quantitative comparisons are important. Our eyes aren't good at comparing colors!
Example: left & right plots have same data & color scale, but colors on the right are binned. Despite strictly less info, the shape is easier to see.
4
9
107
New state of the art for global weather forecasting with machine learning!
📢 Time to share a project I’ve been working on:
Forecasting Global Weather with Graph Neural Networks
🧵 (1/N)
44
309
2K
1
14
108
If only it were this easy! Everytime I've thought this was true, domain scientists have proved me wrong. Machine learning is no shortcut for domain knowledge.
I'm seeing a lot of skepticism from Physicists around my "Learn Physics in 2 Months" curriculum. Machine Learning enables people to make scientific discoveries without needing as much domain knowledge. See my lecture at CERN last year for an example
21
46
318
9
23
101
My first PR to CPython got merged. Yes, it's only a three line doc fix -- but it's exciting to have finally worked my way down to the bottom of the stack!
3
2
94
This makes it all worth it
What did people do before xarray? Not sure where my research would be without it
18
4
136
4
3
90
All that said -- if you're an academic considering industry, don't let "freedom" hold you back.
6
4
90
The secret is a differentiable CFD simulator written in JAX (soon to be open sourced!), which lets us do end-to-end optimization with hard physics priors.
6
1
80
And if you do that, you will soon find yourself accumulating more freedom in your job than you know what to do with.
It's not the same as the freedom of academia. It doesn't come all at once with a grant or tenure. But you earn it all the same, and it can be even more powerful.
1
3
75
xarray v0.10 has been released!
Highlights include:
- indexing with broadcasting over dimensions
- easier wrapping of functions written for NumPy (+ auto-parallelization with dask)
1
53
76
The arguments about symbolic vs deep learning for AI reminds me a lot of arguments about numerical methods vs ML for solving physics problems. Seems like hybrid methods are the way to go in both cases.
Deep Learning Is Hitting a Wall. What would it take for artificial intelligence to make real progress?
#longread
in
@NautilusMag
on one of the key technical questions in AI.
62
248
843
8
3
77
When did everyone forget how hard reinforcement learning is?
I'll believe in AGI when we see chatbots behind the wheel of self-driving cars.
9
0
75
Our latest iteration on benchmarking global weather foreacasts, lead by
@raspstephan
The weather forecast is improving… literally! Introducing WeatherBench 2, a benchmark for the next generation of data-driven, global weather forecast models, providing data, tools, & an evaluation platform. Learn how to use it and check out the website →
18
127
520
0
6
75
The hardest part of research is distinguishing proofs of concept that will actually scale vs perpetual toy examples.
Sadly every attempt I've seen at "intelligence" with neural nets seems to fall in the later category.
5
8
72
Finally, we're open sourced our 2D spectral Navier-Stokes solver with high-order time-stepping in JAX-CFD:
It's differentiable and fast on GPU/TPU, so we hope it may be a useful reference for future ML for PDE efforts, e.g., like
@shoyer
1. A set of standard benchmarks. Compare against high-accuracy baselines (DNS/LES, flow fields should be shared publicly). The spectral code used to compute the flow field should be shared, as well as the CPU/GPU time needed to create the baselines. We DO need the equivalent 2/n
1
0
1
2
18
71
This is perhaps my favorite Xarray release in years.
@cherian_deepak
's Flox library speeds up many common groupby & resample operations by over 10x!
We've just release v2022.06.0!
This release brings a major internal refactor of the indexing functionality (thanks
@benbovy
,
@cziscience
), the use of flox in groupby operations, and experimental support for the new Python Array API standard.
1
28
130
1
10
68
xarray 0.12.2 was released over the weekend:
My favorite new features are the new N-D combine functions by
@TomTomnicholas1
and the ability to append to existing
#Zarr
stores by a team of four (!) contributors, including
@davidbrochart
&
@rabernat
.
1
30
68
I wrote a demo of using JAX's auto-vectorization to implement NumPy style generalized ufuncs. It's pretty sweet!
1
19
63
I can't quite believe what showed up in my inbox today...
I'm not sure whether it's more exploitive to ask students to do unpaid internships (instead, they're charged £500), or open source projects to mentor them without compensation.
4
13
63
This is a nice example from EarthMover of how to tune Zarr data pipelines for loading ML training data into
@xarray_dev
.
Processing Zarr data on the fly is such a better paradigm than trying to anticipate access patterns with a preshuffled dataset on disk.
New blog post from
@_jhamman
: cloud-native data loaders for machine learning using
@zarr_dev
and
@xarray_dev
1/6
3
21
61
1
6
63
Slides for my talk on xarray at the ECMWF Python workshop
#Py4ESS
:
It was excited to hear from so many in the weather data community (
@ECMWF
@bopensolutions
@PyTrollOrg
) about how they're using Xarray & Dask.
0
14
62
The public version of Google's Python style guide has been updated for first time in ~5 years:
1
30
61
@docmilanfar
I don't think you need any of these to be a researcher! Being a researcher means you work on problems that you don't know are solvable by anyone.
1
4
59
Just averaged 52GB of weather data out of core with xray + dask
#pydata
. My laptop has only 16GB of memory.
http://t.co/3OE7ffseyt
1
34
58
I'm really excited about plotnine. It looks like a full featured port of ggplot2 to Python! See also:
2
33
55
Learning the laws of physics: 🥱😴
Learning how to solve the laws of physics: 🤔
Learning how to solve the laws of physics more efficiently than SOTA methods from scientific computing: 😍
1
2
55
For the record: Gemini's insistence on producing diverse images of people is a slightly glitchy feature, not a bug.
If ahistorical images bother you much more than the myriad other generative AI issues, maybe this would be a good opportunity for self-reflection...
4
3
54
To elaborate: I spent most of my 20s in grad school and feeling really dumb, surrounded by people who were better at math & computers than I ever could be.
Maybe this is the natural experience of doing a PhD but it sucked.
@alexeyguzey
Way more. When I was 20, I just wanted to learn interesting things. It took real world experience to realize what I was capable of.
0
0
17
1
0
53
Industry is not a perfect system -- I've only experienced a few corporate cultures, and have undoubtedly benefited from a tremendous amount of privilege. And yes, there absolutely pathological cases where it fails entirely.
1
2
51
Slightly surreal experience with this article. I'm quoted liberally, but I never spoke with this journalist! AFAICT all the quotes are cribbed from a recorded talk that was posted on YouTube.
2
1
48
I’m excited to be part of this effort!
In a Google Keyword blog post, I describe how we are using AI for advanced weather and climate prediction in partnership with
@noaa
and
@ai2enviro
.
0
9
45
1
1
49
@Thomas_ensc
@OpenAI
XLA (which JAX uses under the hood) is great, but it's solving a much higher level optimization problem to generate CUDA kernels from NumPy like code. Triton is lower level and thus offers more control to the programmer, for better or worse.
2
1
48
This is nice work, but improved RMSE on 10+ day deterministic forecasts doesn't mean your model is "substantially better." It means your model is blurrier.
The right baseline is ECMWF's probabilistic ensemble.
Introducing ClimaX, the first foundation model for weather and climate. A fast and accurate one-stop AI solution for a range of atmospheric science tasks.
Paper:
Blog:
Thread🧵
#ML
#Climate
#Weather
#FoundationModel
35
179
850
3
1
49
It's nice to see a paper on efficient neural networks that measures speed on TPU/GPU, not FLOPs! 😍
Revisiting ResNets: Improved Training and Scaling Strategies
- The original ResNets w/ better training + scaling strategies + minor arch change achieve SotA perf w/ faster speed.
- Training and scaling strategies matters more than architectural changes.
2
32
196
0
6
47
Here's an interactive visualization of 728 GB of
@ECMWF
ERA5 temperature data curated by
@pangeo_data
stored in
@zarr_dev
using Neuroglancer.
Be patient: each chunk is 64MB, so it's a little slow to load!
@stardazed0
@zarr_dev
We've been using it to read dataset created with
@xarray_dev
, and it works well! Neuroglancer even knows Xarray's convention for saving dimension names.
Would love to get a
@pangeo_data
demo going on a big climate dataset
0
0
8
1
8
47
Dan writes fantastic and well-documented software. Check it out if you're interested in Gaussian processes!
I'm excited to announce the first beta of a new Python package that I've been working on:
It's (yet another) Gaussian process library in Python, this time built on JAX. It's meant to be both performant & pedagogical with ~4x as many lines of docs as code.
8
51
452
0
3
46
@xarray_dev
@ProjectJupyter
@dask_dev
Also: a special shout-out to
@benbovy
and
@JSignell
for xarray's new HTML repr for notebooks!
It makes it really easy to explore complex datasets (you can click all over to expand/collapse sections). It even wraps
@dask_dev
's HTML repr for showing off nested dask arrays!
2
7
45
The best part is that you maybe be only one short function decorator away from solving your memory problems for good!
If you use Python, take a look at jax.checkpoint, torch.utils.checkpoint or tf.recompute_grad.
I'm sure it's easy in Julia, too -- please reply if you know how!
1
0
45
@jeremyphoward
NumPy uses sorting for set operations, because it doesn't have any hashtable data structures. So np.isin(a, b) is OK for a large a and small b, but not the other way around.
I agree that it would be good to have some performance warnings in the docs.
2
4
42
Now flying to Austin for
#SciPy2018
. I'm looking forward to productive discussions about
#NumPy
,
#pandas
,
#dask
,
#xarray
and
@pangeo_data
!
2
1
45
I'm excited to share my work with
@yohaibs
on using differentiable programming to learn better discretizations for PDEs, now published in PNAS.
New research shows how
#machinelearning
can improve high-performance computing for solving partial differential equations, with potential applications that range from modeling
#climatechange
to simulating fusion reactions. Learn all about it here ↓
5
186
501
4
5
45
I will be at AGU next week. If you want to talk about ML-based weather/climate modeling (NeuralGCM, WeatherBench2, etc) or
@xarray_dev
please reach out!
1
0
45
2
1
43
I'll be presenting this work with
@samgreydanus
and
@jaschasd
tomorrow morning (Friday Dec 13) at 11:30am at the NeurIPS Deep Inverse Problems workshop ()
I'm happy to share a new paper, with
@jaschasd
and
@samgreydanus
: "Neural reparameterization improves structural optimization"
We use neural nets to parameterize inputs of a finite elements method, and differentiable through the whole thing:
2
38
170
2
9
42
To clarify, the likely focus is on fluid dynamics problems in climate & weather using hybrid physics/ML techniques:
0
2
43
@chrmanning
There is undoubtedly loads of compute wasted on supercomputers, but simple AI shortcuts (like in this paper) are far from ready to replace physics based simulations. They don’t generalize in any meaningful way.
2
6
41
Interesting note: by far the most expensive part of training these sorts of models is *validation*. We have to run reference solvers on 32x higher resolution grids in space + time in order to rigorously measure the accuracy of our ML solver.
4
0
38
But as I learned from Tim Palmer's delightful book "The primacy of doubt" the situation for weather forecasting is actually far more dire: as Lorenz showed in 1969, each doubling of resolution only gives half previous gains in accuracy.
1
1
40
Unpopular opinion: rainbow colormaps can be an excellent way to visualize scientific data!
My favorite is "turbo:"
10
1
38
@jaschasd
@samgreydanus
I'm happy to share:
(1) this paper was accepted for an oral presentation at the NeurIPS 2019 Deep Inverse workshop!
(2) we're released the source code, so you can now run it yourself!
1
6
38
@bilalmahmood
Bilal, I supported your campaign, but this is not a good look!
You should never had led a campaign ad by calling yourself a "neuroscientist" based on a year or two of part time lab experience as part of your undergrad degree. It diminishes the credentials of real scientists.
3
2
38
Love this paper!
A nice example of when gradients have questionable value in geoscience is optimizing parameters of a climate model.
New paper: when to use gradients
DL researchers often compute derivatives though just about everything (physics simulators, optimization procedures, renderers). Sometimes these gradients are useful, other times they are not.
We explore why.
1/7
13
158
797
1
1
38
Hitting refresh in TensorBoard is the ML equivalent of pulling the lever on a slot machine. It's every bit as addictive, but possibly even more expensive!
4
0
37
I'm sad to leave my colleagues at
@ClimateCorp
but stoked to be joining
@GoogleResearch
today!
12
0
38
I designed the
@xarray_dev
logo in TikZ 🙈
interviewer: can you explain this gap in your CV
me: yeah I was trying to make a complicated figure in TikZ and lost track of time and—
interviewer: say no more
10
142
2K
2
1
38
One of my regrets with
@xarray_dev
is that I copied end-inclusive slicing rules from pandas. At this point there's basically no way to fix it without silently breaking lots of code.
I should say that Pandas is a great resource for the Python community and enables my research. I just wish it didn't have such a strong "personality." Its own slicing rules (end of range is included), its own membership function (isin), etc.
3
1
25
2
1
37
If I was rewriting NumPy's API from scratch today, I would be tempted to make both array indexing and reductions over specific axes preserve rank, i.e.,
1. `x[i]` would be equivalent to `x[i:i+1]`
2. `x.sum(axis=0)` would be equivalent to `x.sum(axis=0, keepdims=True)`
4
1
36
Maybe I'm the problem: I want my work to influence domain scientists (and get sensible reviews), so I rarely submit to ML conferences (and thus don't get asked to review). 🤷
2
1
36
Fun bonus fact: data assimilation was one of the first use-cases motivating the development of auto-diff software.
E.g., see this 1993 paper
So using modern libraries like JAX for data assimilation is really going back to the roots of the field :)
0
1
35