Tim Duignan
@TimothyDuignan
Followers
2,065
Following
1,722
Media
140
Statuses
2,524
Modelling and simulation of electrolyte solutions using quantum chemistry, stat mech and neural network potentials #compchem #theochem
Brisbane, Queensland
Joined February 2013
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
#RightPlaceWrongPerson
• 633081 Tweets
namjoon
• 371950 Tweets
Doge
• 184213 Tweets
RCB FINISHED DHOBI
• 159151 Tweets
How Sweet
• 141025 Tweets
LOST OUT NOW
• 118066 Tweets
新時代の扉
• 72116 Tweets
Clancy
• 71602 Tweets
Kabosu
• 58193 Tweets
#ThinkDeeply
• 41268 Tweets
Hayırlı Cumalar
• 33898 Tweets
#يوم_الجمعه
• 28510 Tweets
メラルバ
• 25081 Tweets
夏アニメ化
• 24104 Tweets
エアコン
• 22101 Tweets
ウマ娘の映画
• 21531 Tweets
移民政策
• 20159 Tweets
スタライ
• 19808 Tweets
Last Seen Profiles
Pinned Tweet
Tried to capture why I’m so excited about NNPs. I honestly think they are a profoundly useful tool.
The Potential of
#Neural
Network Potentials
A perspective from Timothy Duignan
@TimothyDuignan
@Griffith_Uni
🔓 Open access in ACS Physical Chemistry Au 👉
1
30
166
7
14
117
I think neural network potentials are the most important scientific tool of the next decade. The ability to simulate systems at the molecular scale starting from nothing but quantum mechanics will be transformative for a vast range of problems throughout biology and chemistry 1/n
The Potential of
#Neural
Network Potentials
A perspective from Timothy Duignan
@TimothyDuignan
@Griffith_Uni
🔓 Open access in ACS Physical Chemistry Au 👉
1
30
166
9
91
615
I want to explain a statistical mechanical concept known as coarse graining which I think might be useful for thinking about things like AF3. Especially a special case known as continuum or implicit solvent models.
Thrilled to announce AlphaFold 3 which can predict the structures and interactions of nearly all of life’s molecules with state-of-the-art accuracy including proteins, DNA and RNA. Biology is a complex dynamical system so modeling interactions is crucial
186
2K
8K
7
39
322
Ok so the new AlphaFold model relies in large part on a "relatively standard diffusion approach" turns out you can think of this as just a special case of a neural network potential, it just uses experimental data not quantum chemistry to train on. 1/n
Thrilled to announce AlphaFold 3 which can predict the structures and interactions of nearly all of life’s molecules with state-of-the-art accuracy including proteins, DNA and RNA. Biology is a complex dynamical system so modeling interactions is crucial
186
2K
8K
6
41
306
Another very impressive general purpose graph NN for molecular simulation. Feels like this field is really accelerating.
4
39
214
Ok so what is a neural network potential concretely? It's just a very flexible function with many adjustable parameters that you fit to the 'potential energy surface.' This is just the energy as function of the position of the atoms in your system. 1/n
I think neural network potentials are the most important scientific tool of the next decade. The ability to simulate systems at the molecular scale starting from nothing but quantum mechanics will be transformative for a vast range of problems throughout biology and chemistry 1/n
9
91
615
5
33
205
Wow this blows me away you can get roughly correct structure of pure electrolyte solutions out of this.
Thrilled to announce AlphaFold 3 which can predict the structures and interactions of nearly all of life’s molecules with state-of-the-art accuracy including proteins, DNA and RNA. Biology is a complex dynamical system so modeling interactions is crucial
186
2K
8K
10
27
202
Clearest explanation of this beautiful idea that I've found so far.
2
22
172
Is there a machine/deep learning textbook anywhere that teaches the Boltzmann/Gibbs distribution? Is it called something else? Have looked at three so far with no mention of it. It is the entropy maximising distribution! Surely it is important to know?
27
16
126
Interesting how deep learning for generating equilibrium distributions seems to be converging back to molecular dynamics. Like this is just langevin dynamics with a learnt score. So just NNP-MD with many runs in parallel right? Or am I missing something?
Predicting equilibrium conformations in protein via
#ML
, what else soon, to predict non-equilibrium distributions. Thoughts?
2
10
66
8
11
126
@LakeBrenden
@jeremyphoward
ChatGPT’s response is lot better. The last line is a little suspicious though.
5
1
92
So pleased to get this preprint out. Feel like we’ve finally worked out how to do something I’ve been trying to do for 13 years since the start of my PhD: Build an accurate continuum solvent model of ion-ion interactions in solution.
5
11
83
I want to record a prediction: ML acceleration of molecular simulation will transform all of physical science. From quantum scale all the way up to climate. Justification: 1/n
Yes, this is the ultimate way ML will help accelerate physical sciences. By constructing custom MCMC operators (eg proposal distributions) to accelerate traditional MD/MCMC simulations in combination with existing tools. This can be done while preserving all error bars.
3
11
120
3
3
80
Amazing how far ahead Behler and Parrinello were. There will be a Nobel prize for this paper one day imo:
3
5
65
Quantum computing experts claim computing properties of Femoco is impossible with classical computing and if you could do it you could revolutionize fertilizer synthesis. Turns out you can do it with DFT fine but almost no one cares.
6
2
58
Ok for all the haters out there I managed to break it which was what I was originally trying to do/expecting. Here's NaCl. 🤣
Wow this blows me away you can get roughly correct structure of pure electrolyte solutions out of this.
10
27
202
7
5
53
Love the flow of ideas back and forth between molecular simulation and deep learning. Diffusion models originally inspired by molecular dynamics algorithms (langevin dynamics) now inspiring new approaches to accelerate MD.
Oke, the AlphaFlow paper is awesome: AlphaFold Meets Flow Matching for Generating Protein Ensembles
Just watch how AlphaFlow's ensemble reproduces details of MD.
Weights + code
We have it in the reading group on Mon 11am EST!
1/2
15
152
760
2
10
50
Notice how similar to MD this is conceptually. It is actually mathematically essentially the same also. The only difference is the force field is learnt from the PDB where you know the forces are 0 because they are equilibrium states. Really its an implicit solvent force field.
RFdiffusionAA generating a small molecule binding protein against an experimental FXIa inhibitor (OQO), a ligand which is significantly different than any in its training dataset.
1
62
244
4
5
44
Some people are not impressed by this. Maybe im just incompetent but I spent literally years trying to build continuum solvent models of this exact thing and couldn’t do much better it’s really hard to model without explicit water! 🤣
Wow this blows me away you can get roughly correct structure of pure electrolyte solutions out of this.
10
27
202
4
0
42
We see exactly the same thing for simple electrolytes. If you cannot get sodium chloride pairing free energy right you are not going to get protein folding right. I often don’t point this out because I don’t want to offend senior researchers.
Take a look at this
#OpenAccess
paper 📝 from the latest issue of Journal of Chemical Theory and Computation
#JCTC
🔎 The Role of Force Fields and Water Models in Protein Folding and Unfolding Dynamics 💦🔬
🔓
#thermodynamics
0
7
19
6
3
38
This is a beautiful clear explanation of diffusion models. The cool thing is they are actually really easy to understand if you know molecular simulation. There is a direct analog for almost every concept. 1/n
New blog post about the geometry of diffusion guidance:
This complements my previous blog post on the topic of guidance, but it has a lot of diagrams which I was too lazy to draw back then! Guest-starring Bundle, the cutest bunny in ML 🐇
9
77
355
1
5
38
Good take as always. I don’t think this axis makes sense really though. I would argue a diffusion model is more physics based than a lennard jones forcefield. Harmonic approximation about the minima is in every physics text book but I’ve never seen a 1/r^12 repulsion.
5
4
37
Awesome paper. Shows how we can train on many different levels of theory simultaneously will be very important as we make DFT databases bigger and bigger. We need to build a PDB equivalent but for quantum chemistry.
1
0
35
Another nice ion pairing paper on NaCl with NNPs. Look at the spread on those classical force fields in comparison! This is the fundamental medium in which all of biology occurs and we haven't been able to predict even its most basic properties until now!
1
6
35
So byte dance have entered the universal machine learned force field race with a very impressive paper starting with the right problem imo: liquid electrolytes. I think this could be a critically important technology.
1
7
34
Just imagine one day we will be able to go to a website like this and run accurate dynamics on any system of atoms we want. This will transform all of science and society. We will finally be masters of the molecular scale.
1
2
34
So cool! I assume this is the same thing that goes on at phase transition boundaries in stat mech: ‘Schramm-Loewner curves appear as domain boundaries between phases at second-order critical points like the critical Ising model’
Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.
279
2K
10K
2
4
32
Very cool. Diffusion models use a molecular simulation algorithm (thermally annealed langevin dynamics) so of course you see phase transitions directly analogous to the sudden changes that occur when you cool/heat a system of molecules, i.e., crystallization.
A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
1
32
151
2
2
30
This means that Google's claim that they have "surpassed physics based tools" is kind of strange. In fact there is a ton of physics baked into it how diffusion models work!
1
4
31
Check out this really nice collaboration with
@alisterpage
and two awesome students where we show you can resample from DFTB MD, compute forces at a higher level of theory and run stable MD with equivariant neural network potentials.
1
5
30
Fascinating, maybe we should find the right ion force field parameters before we spend another $42.5 billion on failed drug trials?
1
3
28
Fascinating summary of recent work on geometric/graphs neural networks. I’m catching up on this field. But I’m now convinced it will change the way we do a lot of science. In particular I really like
@HannesStaerk
’s point re. the application to learning quantum interactions. 1/7
An annual round of predictions in Geometric and Graph ML coauthored with
@PetarV_93
based on input from the leading experts in the field.
Longread on
@TDataScience
3
77
267
1
5
27
If you trained on the true equilibrium distribution structures extracted from a simulation you get the true forces (if your noise is sufficiently low) This paper first showed this and we have validated for a simple system that it is precise.
1
4
27
Neural network potentials trained on high level quantum chemistry calculations is the only plausible solution to this problem imo. Is there an alternative?
despite advances in protein folding models like AlphaFold, we haven't actually discovered that much new about the underlying principles of protein folding.
these models accurately predict the structure a given protein sequence will fold into, without knowing that much about the
5
24
166
5
0
22
So Mg and Ca look plausible but don't actually form pairs with chloride in water at all in reality so not really physically correct. On the other hand classical simulation approaches can also fail to get that right. The fact they're all roughly the same peak height indicates
Wow this blows me away you can get roughly correct structure of pure electrolyte solutions out of this.
10
27
202
1
1
22
This is what I was looking for. I derived this independently and couldn’t work out why I hadn’t seen it before. I think this is a profoundly beautiful and important result. Learning is isomorphic to statistical mechanics!
4
1
22
Next I'll do a thread on where to now that the PDB has been tapped out. Google say they're going to wait for 'cryo electron microscopy and tomography' to give them more data. This will take too long and is inherently limited to equilibria there is a faster way in my opinion.
5
0
22
Wow very cool thread. Amazing it can get this stuff right to me when it doesn’t even know about electrostatics. Thread shows some interesting ways it breaks too though.
I read a lovely post on how AF3 can predict electrolyte RDFs. So I digress from protein-NA complexes to just NA complexes. (Will come back to it)
Case 3: 1JRN oxytricha bimolecular G4T4G4. My fav since I solved it.
Perfect prediction with loops and 5 ions.
1
3
25
0
6
21
I assume someone’s now furiously coding up a KAN for modeling potential energy surfaces? Good luck!
MLPs are so foundational, but are there alternatives? MLPs place activation functions on neurons, but can we instead place (learnable) activation functions on weights? Yes, we KAN! We propose Kolmogorov-Arnold Networks (KAN), which are more accurate and interpretable than MLPs.🧵
121
1K
5K
2
0
21
Kind of hoping we get to use these huge gpu clusters to run massive molecular dynamics simulations.
The biggest winners of AI/GPU gold rush will be those who figure out what to do with all these shovels once it becomes apparent there is no gold.
22
39
321
3
1
21
Lots of exciting new machine learning for molecular simulation papers coming out: one arguing that equivariant features are key and should become standard:
1
0
20
These are already being used today to design new drugs. And everyday they get much better. They work by predicting the solution of the Schrödinger equation much faster than it’s possible to directly solve it.
1
3
20
@ccanonne_
This is nice but a problem is it’s getting better at spotting it’s own mistakes!
Instead of finding the perfect prompt for an LLM (let's think step by step), you can ask LLMs to critique their outputs and immediately fix their own mistakes. Here's a fun example:
61
226
2K
3
0
18
And in fact diffusion models were explicitly inspired by statistical mechanics.
1
2
19
I think this is more or less correct. The only limitation is training data. Where will that data come from?: AI accelerated first principles molecular simulation will be a big source in my opinion.
"Where do I think the next amazing revolution is going to come? And this is going to be flat out one of the biggest ones ever.
There's no question that digital biology is going to be it."
Jensen Huang, founder & CEO of NVIDIA.
31
186
926
1
1
18
@StasBekman
This is a phase transition we see it in molecular simulations which are directly analogous. Loss: energy, entropy: int rho log rho: neural nets minimise the free energy so you can see states with similar free energy but different losses and jump between.
1
0
19
They predict the forces on atoms allowing us to simulate how atoms and molecules move. They therefore connect the quantum scale to the classical scale. But as important an achievement as that is they are even more useful than that.
1
2
17
Amazing how something as well studied as a protein folding can actually take up to 1000 times longer than we previously thought. Crazy to me how much we still don’t know about the molecular scale.
1
4
17
In contrast diffusion models learn the "score" which is the gradient of the log of a probability distribution. For AF3 this is just the probability of the atoms having a particular position. In stat mech log probs are free energies or potentials of mean force and their grads are
1
3
17
Love all the AI guys talking about thinking from ‘first principles.’ They should follow that thinking through and pick up a text book on the real first principles: the principles of quantum mechanics.
4
2
15
Ah this makes sense. The equivariance is introduced through data augmentation this is a well established idea.
1
0
16
This is because the problem of connecting scales is much more general than just the quantum to the classical there’s also connecting the scale of molecules to proteins and proteins to cells and cells to organs and so on. Same in chemical engineering and climate simulations.
1
1
16
It should be possible to automate this process as we know from renormalisation group theory that there are recurring mathematical features involved in connecting scales.
1
2
15
Excellent point:
Excited to share our perspective paper
@JPhysChem
"
#MachineLearning
Interatomic Potentials and Long-Range Physics"
#compchem
It's focused methodologies & models used where presence of nonlocal physics & chemistry phenomena for molecular properties
1
20
91
1
2
15
This work nicely demonstrates why equivariance is such a game changer for NNPs. Dramatically increases generalizability.
Interested in water simulations and/or machine learning? We show how to develop transferable water potentials using equivariant neural networks!
4
14
73
0
3
14
Not only are denoising diffusion models incredibly powerful, the maths behind them is very cool! They are effectively learning to do gradient ascent using gradient descent, kind of a meta gradient descent. There's also a great connection with comp. chem.
3
1
15
Neural network potentials are already enabling this. (Diffusion models and Alphafold can also be interpreted as more general examples of neural network potentials) We should soon be able to accurately simulate the intermediate scale processes smaller than we can observe directly.
2
2
15
I'm incredibly thankful and excited to say that I've been awarded an
@arc_gov_au
#DECRA
fellowship based at
@UQ_News
to work on discovering new electrolyte solutions for energy storage applications!
5
0
15
This is related to diffusion models as you can show that if you train a diffusion model on equilibrium structures (not just minima) with low nose the score you learn corresponds to the true mean forces and you’re therefore implicitly learning the actual free energy surface.
2
1
15
Wow source coding theorem is awesome. Why can you store a crystal structure in a small file vs a liquid which needs a very big data file? One has much higher physical entropy and this needs more data! That intertwining of physical
and information theoretic entropy is beautiful
3
2
15
Phase 3 is the real bottle neck so maybe too early to tell though
0
0
1
Just read this excellent paper from
@ixfoduap
. This is a really useful tool! Developing optimizable DFT functions is absolutely critical for simulating important realistic systems.
1
1
15
I think this is a profound paper … This is what ‘grokking’ is right? A sharp jump downward in energy/loss? It’s just a phase transition right? Stat. mech. must have the tools to explain the success of deep neural networks.
1
1
14
Seems like ML is playing a similar role in climate physics as it is in molecular scale physics. My dream is for someone to build a first principles based global climate simulation using iterative coarse graining. Is that ridiculous?
Finally finished a new review paper: "Machine learning for climate physics and simulations"
@turbulentjet
We highlight the distinct yet complementary goals of ML: accelerating simulations vs learning physics. Share with us your favorite ML4climate papers!
5
112
442
1
1
14
This approximation of linear forces back to minima has a long history in physics its called the harmonic approximation and physicist famously use it everywhere. It results in a Gaussian probability distribution and a nice smooth surface to optimise on. This paper outlines this
1
2
13
Another incredibly impressive electrolyte simulation paper with machine learning potentials. Still many potential improvements though too. Soon we will be able to predict almost everything you could want to know about a given electrolyte with no experiment necessary.
2
0
14
In particular at each scale there is the problem of ignoring the fast dynamics that can be ignored or approximated with Gaussian noise and keeping track of the important features that are useful for prediction. Machine learning is the perfect tool for this.
1
1
14
This is why I think simulating liquids is a perfect application of machine learning to science. It’s a problem where you need to do a ton of inference mostly in domain very fast ie the energies of every frame of a simulation.
💯
Scientific discovery is about at the tails of existing knowledge/data, machine learning is about the bulk of the existing data.
Doesn't mean the latter cannot assist with the former, but it's highly non trivial.
0
16
115
1
2
12
This was enabled entirely by the many amazing software packages people have built, especially: CP2K, NequIP and LAMMPS.
0
3
12
Half of my time line is people saying AI is just curve fitting anti science. The other half is righty freaking out cause it can do stuff like this. I’m so confused.
4
1
13
Exciting times: we have the accuracy with quantum chemistry and the speed with ML that we need to really start cooking with computational chemistry for many condensed phase applications. From: Industry needs to invest in this.
4
0
13
First off a neural network potential is just a very flexible function with a ton of parameters that takes in positions and outputs energies/forces. More detail in this thread:
Ok so what is a neural network potential concretely? It's just a very flexible function with many adjustable parameters that you fit to the 'potential energy surface.' This is just the energy as function of the position of the atoms in your system. 1/n
5
33
205
1
3
13
Twitter is still such an amazing search engine! So much to read now.
Is there a machine/deep learning textbook anywhere that teaches the Boltzmann/Gibbs distribution? Is it called something else? Have looked at three so far with no mention of it. It is the entropy maximising distribution! Surely it is important to know?
27
16
126
0
0
12
We can’t rely on bespoke tools hand crafted for connecting each of these scales. That would take too long to build we need a general purpose solution.
1
2
12
Excellent piece. Explains why combining first principles methods with ML is the way to go to have an impact in chemistry.
1
1
13
I actually think this is a great example of why AI is so important. This is a relatively simple combination of ions and solvents that has this remarkable ability and yet it has taken us decades to discover it.
1
1
13
Machine learning seems to have had a much less dramatic impact on direct quantum chemistry than I expected. Maybe it’s still early days but maybe also it’s because a lot of the algorithms are already so similar to ml algorithms that there is not much to be gained.
6
0
12
Another trick diffusion models use? They start with a high noise level and then gradually reduce it to refine the distribution. This is essentially thermal annealing another tool from stat mech.
1
1
13
100%
"The pressures facing today's young research scientists makes it hard for them to find the time simply to think" - Roger Penrose, in this first-rate profile from
@philipcball
7
77
371
1
0
12
So when the diffusion model learns the score it is implicitly learning a free energy and its gradient essentially making it a type of NNP. Now you may object that AF3 doesn't know anything about the forces or the energies as it is trained on the PDB so how can it possible be
1
2
11
And a final trick? They use Langevin dynamics for inference. A standard molecular simulation algorithm invented by a Physicist. When you run Langevin dynamics you get Boltzmann probabilities. i,e,. exponential of the free energy. So we end up back with original probabilities.
1
1
11
This is a fascinating and excellent piece. IMO this part might be a bit pessimistic though. Extending quantum chemistry to larger and longer scales is actually a problem perfectly suited to ML using exactly the same tool that makes alphafold work: equivariant NNPs.
In a new post with
@NaefLuca
@vant_ai
we ask what is needed for ML to succeed in biological problems and argue that it is necessary to move beyond "white-box data"
w inputs from
@nathanbenaich
@dom_beaini
@befcorr
@georg_e_winter
@schwabpa
@ggrigoryanv
8
80
287
4
2
11
Neural network potentials can make these predictions thousands of times faster than possible with direct solution of Schrödinger equation.
1
1
11
Predicting energies from coordinates is much more general than just simulating atoms directly though. We often want to ignore parts of the system (marginalise) and this means we want to calculate free eneriges which again determine the probabilities of particular arrangements.
1
1
11
One I’m working on now is simply the prediction of the thermodynamic properties of electrolyte solutions. We can easily generate large high quality data sets of properties of these solutions from QM and it should be possible to predict their properties from that with GNN. 5/7
1
1
11
This is an essentially identical problem but can work for much bigger particles. You just train them to learn the average forces of a subset of your particles. (Free energies are just determined by the average forces which is a very nice stat mech trick).
1
1
11
For a single particle in 2D you can visualise this concretely as a real surface where the height corresponds to the potential energy. Mostly we care about hugely high dimensional versions of this though where you have many particles, so you have:
1
1
11
Yeah very important work. This is the key issue preventing progress now: Force field dependence. We can overcome this with machine learning potentials and good DFT now.
0
0
11
This makes sense to me. It goes against the bitter lesson but I think most problems in bio don’t have enough data for that to apply?
To all budding compbio & ML folks interested in bio: Don't just only run behind the latest ML model hype train. The greatest long run impact will come by really assimilating prior bio/compbio literature with the goal of really understanding strategies for how to model biology. 1/
8
129
784
3
0
10
Why is it called a PMF? Because you can show that its derivative with position is equal the average forces in a given configuration. (Nice proof to try yourself) So this allows you to connect microscopic forces to marginalised probabilities which is very useful.
1
0
10
Ha literally just gave a talk where I made this same point. Diffusion models are best understood from an equilibrium stat mech pov even though they are inspired by non equilibrium. Was a bit weird cause it was a NE tat mech workshop.
The statistical thermodynamics of generative diffusion models. (arXiv:2310.17467v1 [])
1
39
177
1
0
9
Equivariance is very cool and draws on some deep mathematics that has already revolutionised theoretical physics. But the intuition is just to use neural networks that can keep track of, and compare, directions not just raw numbers.
1
1
10