I value approaches that work by subtraction: stripping away the unnecessary until the essential insight remains. Doing less often demands more: a deeper understanding of data and method to reveal the simplest formulation at the core. This is an attempt of ours. Hope it resonates.

6/ Our results demonstrate that many favorable properties of GNNs can emerge adaptively and more flexibly in Transformers, challenging the necessity of hard-coded graph inductive biases and pointing toward standardized, scalable architectures for molecular modeling. This has

5/ We really do mean an unmodified Transformer: no explicit calculation of pairwise distances, no graph-based features, no rotational equivariance, etc. Leveraging modern software and hardware, a 1B parameter Transformer trains and runs inference faster than a 6M parameter

4/ The model learns attention scores that decay inversely with interatomic distance in early layers, and in later layers learns attention that remains roughly constant with distance. This suggests that the Transformer naturally first learns local feature extraction before

3/ The model predictably scales with training resources, with no signs of saturating!

2/ A Transformer can adaptively learn attention patterns based on atomic environment, whereas GNNs often use a fixed graph construction algorithm (e.g., k-nearest neighbors or fixed radius cutoff)

1/ Can molecular AI move past hard-coded Graph Neural Networks and embrace scalable Transformers that discover molecular structure on their own? We demonstrate that you can train a 1B parameter Transformer model without any graph priors or physical inductive biases. And

9/ This was a very fun project to work on with an amazing team (Sanjeev Raja, Martin Sipka, Michael Psenka, Toby Kreiman, Michal Pavelka) and a great way to explore statistical physics + generative modeling connections! Paper:

8/ Our method is a way to turn any score-based generative model of IID molecular configurations into an efficient hypothesis generator for dynamical events, without system-specific training. We're excited to advance this method as generative models continue to scale and improve!

7/ We show that the path ensemble sampled by our OM optimization approach can be used to compute transition rates. Specifically, we can estimate the committor function by minimizing a well-known variational objective over the sampled paths, from which we can compute the rate

6/ We show examples where our method generalizes beyond the original training data, predicting accurate transition paths in systems not seen by the generative model, and when data from the transition regions is selectively removed

5/ Results on protein folding transitions show our method efficiently generates diverse, physically meaningful pathways, closely matching reference molecular dynamics (MD) simulations

4/ The inherent stochasticity of generative models can be used to efficiently produce many diverse initial guesses for the transition path. Each initial guess relaxes into its nearby local minimum during OM optimization, allowing us to approximate the transition path ensemble

3/ We interpret transition paths as realizations of stochastic dynamics induced by pre-trained diffusion and flow matching models. This turns path sampling into an OM action-minimization problem, with the learned score function being a "force field" over the data distribution

2/ Our approach leverages least action principles by minimizing the Onsager-Machlup (OM) functional. Normally, this approach can only find a single minimum-energy path; this is where generative models come in handy! :)

1/ Generating transition pathways (e.g., folded ↔ unfolded protein) is a huge challenge: we tackle this by combining the scalability of pre-trained, score-based generative models and statistical mechanics insights-—no training required! To appear at #ICML2025

Excited to see the OMol25 dataset out! This was really fun to collaborate on 😃

A knowledge distillation approach to get fast, specialized machine learning force fields: this work will be presented at #ICLR2025 this Sat (Apr 26, 10 AM) at Poster Session 5 + it will also be a spotlight talk at the #AI4Mat workshop on Apr 28!

7/ This was a very fun project with Ishan Amin and Sanjeev Raja, who will be presenting this at #ICLR2025! Paper and code below: Paper: https://t.co/lXbvyHXw47 Code:

6/ The distilled MLFFs are much faster to run than the original large-scale MLFF: not everyone has the GPU resources to use big models and many scientists only care about studying specific systems (w/ the correct physics!). This is a way to get the best of all worlds!