Our new paper in @NatMachIntell tells a story about how, and why, ML methods for solving PDEs do not work as well as advertised. We find that two reproducibility issues are widespread. As a result, we conclude that ML-for-PDE solving has reached overly optimistic conclusions.

RT @anderssandberg: If this is correct, the Illusion of Thinking paper will really drop in my esteem.

RT @NicolasRasmont: We just published a post-mortem on a now-retracted viral AI‐materials paper from MIT. Graduate student Aidan Toner-Rod….

RT @pli_cachete: American funding for hard sciences has fallen 2/3 this year. In physics, they are receiving 15% of what they did last yea….

RT @KordingLab: AI for science appears hard. Here is my stance on AI in science: AI is a great side-kick. I am unconvinced it is time to ma….

In a guest post for Understanding AI (@binarybits), I write about how I got fooled by AI-for-science hype, and what it taught me. I argue that AI is unlikely to revolutionize science, and much more likely to be a normal tool of incremental, uneven scientific progress.

I'm looking forward to speaking at the AI summit in Tokyo in 2 weeks. 2週間後に東京で開催されるAIサミットで講演できることを楽しみにしています。.

In other words, if a scientist tries using machine learning for a "real scientific problem" similar to the ones explored here (i.e., spatiotemporal data), most of the time they'll find that ML is worse than useless!. And even in the 29% of cases where the ML model does better.

As the authors readily admit, these models aren't state of the art. With enough effort and tuning, ML *could* do better than this. But they use "time-tested models that are widely used in applications", and "reflect reasonable compute budgets and off-the-shelf choices that might.

The most interesting part of this paper is how poorly ML does at scientific problems. See table 3. Four "popular models" are trained on 17 datasets. Out of 128 total evaluations, ML does worse than the weakest possible baseline (outputting a constant value) 71% of the time.

RT @RickSteves: I miss the Grand Old Party (GOP) and long for the day when we can return to healthy and principled partisan debates. (We id….

Bram van Leer should win a Nobel Prize. His 5-part series of papers in the 70s laid the foundation for modern CFD.

RT @LorenaABarba: I addressed the topic in my keynote at PASC more than a year ago, but of course Nick's paper now gives us solid evidence….

RT @shoyer: A nice summary of why I moved on from ML for PDEs. The literature on weather & climate modeling isn't perfect, but the baseline….

While the causes of these issues are complex, the most important factor appears to be a systemic bias in ML research towards positive results. To read the full discussion, use the open-access link below.

The second issue is reporting biases, especially outcome reporting bias and publication bias. Weak baselines lead to overly positive results. Reporting biases lead to underreporting of negative results. The end result is overoptimism about ML.

The first issue is comparing with a weak baseline. We perform a systematic review of this research area, focusing on PDEs relevant to fluid mechanics, and determine that 79% (60/76) of papers compared to a weak baseline.

While this strategy is quite general, the specific error-correcting algorithms need to be tailored to the invariants of the underlying PDE. The bulk of the paper is dedicated to deriving, understanding, and demonstrating the consequences of these algorithms.