🚨New paper! Generative models are often “miscalibrated”. We calibrate diffusion models, LLMs, and more to meet desired distributional properties. E.g. we finetune protein models to better match the diversity of natural proteins. https://t.co/2c06vD0x2D https://t.co/9Tbhf6ml8K

Super excited to share this preprint with @nate_diamant and my advisor @brianltrippe on how we can fine-tune diffusion models, language models, and more to match known distributional properties. Check it out!👇

This is a first paper from my group, joint with @smithhenryd and @nate_diamant . Paper: https://t.co/2c06vD0x2D Code: https://t.co/9Tbhf6ml8K If you’re interested to try it out on a new problem together, or have ideas for how to improve the method, please reach out!

For language models: CGM reduces profession-specific gender bias 5× more than prior methods. Bias reduction generalizes somewhat to held-out professions – though model extrapolation remains a challenge.

For image models: CGM fixes severe class imbalance in conditional generation (e.g., more foxes, fewer lions 🦊🦁). Samples stay realistic.

Applied to protein design: Previously, inference-time heuristics traded-off between diversity and quality. CGM increases structural diversity to match proteins from nature, without sacrificing designability!

Two algorithms: – CGM-relax: add a miscalibration penalty – CGM-reward: turns calibration into reward fine-tuning They work on diffusion models (continuous & discrete!), LLMs (autoregressive & masked!), normalizing flows, and more. Only sampling and model likelihoods are needed.

We frame calibration as a constrained optimization problem: Finetune towards the model closest in KL-divergence to the base model that satisfies a set of calibration constraints.

Why does this matter? – Protein design models miss structural diversity – LLMs mirror social biases present in training data – Image models can suffer mode collapse Calibration errors limit fairness and scientific utility, and aren’t solved by finetuning on sample-level rewards.

Super excited about this progress and next steps! Come chat with us at our poster tomorrow morning if you’re at #ICML2025

Actually happening at **11 AM ET** !

Interested in functional protein design or motif-scaffolding? We propose a standardized protein design benchmark, MotifBench aimed at tackling key challenges in the field. Preprint: https://t.co/3SsdPk8p0c Code: https://t.co/VoF3zdd7BN (1/N)

Have a look at our shiny new benchmark for motif-scaffolding in computational protein design! New (and harder) tasks, including a reproducible evaluation pipeline

We made a new, reproducible, fair, (and much harder) motif scaffolding benchmark! With Zhuoqi Zheng, Bo Zhang, @DidiKieran @json_yim @_JosephWatson Hai-Feng Chen, @brianltrippe

And thanks as well to ChatGPT for turning my first banal attempt at a tweet-thread into a emoji-packed click-bait for real work! 😂

This has been a big team effort – with contributions across 5 timezones 🌍. This is thanks to my fantastic coauthors on our whitepaper:

This is a V0 pilot—we need your input for V1! 🧩 Know of an important motif? Add it to the benchmark! 🏗️ Help improve the pipeline & metrics (sequence-based? Side-chain-level?) Let’s shape the future of motif scaffolding together!

Surprise: A modern baseline (RFdiffusion) fails on motifs scaffolded into de novo enzymes 15 years ago 🤯 This suggests modern deep learning methods aren’t always better than past methods, and there is much room for progress!

MotifBench fixes this by providing: 🧪 A standardized evaluation pipeline 🏆 30 challenging motifs as test cases 📊 Easy-to-use eval scripts and a leaderboard for method comparison Now, results can be easily and consistently measured.

Recent progress in motif scaffolding has been exciting! But… ❌ Current evaluation are inconsistent, and results incomparable ❌ Widely used test cases are too easy ❌ Reproducibility is difficult That’s where MotifBench comes in.