Authors: @gujral_onkar, @mihirbafna14, @ejalm and @lab_berger.

8/ Paper: MIT News article: Github:

Introducing Grok Imagine.

7/ We show that SAE & transcoder features are much more interpretable than ESM neurons, for both protein-level & amino acid-level representations. This has the potential to improve safety, trust & explainability of PLMs. As PLMs improve, SAEs could help us learn new biology.

6/ We also use Claude to autointerpret SAE features based on protein names, families, gene names & GO terms. Many features correspond to families (like NAD Kinase, IUNH, PTH) & functions (like methyltransferase activity, olfactory/gustatory perception).

5/ We interpret these SAE features using Gene Ontology (GO) enrichment. Many protein-level SAE features align tightly with GO terms across all levels of the GO hierarchy.

4/ SAEs have a very wide latent dimension with a sparsity constraint. This forces PLM representations to disentangle into biologically interpretable, sparsely activating features without any supervision.

3/ We train sparse autoencoders (SAEs) on protein-level and amino acid-level representations from layers 6-10 of ESM2_t12_35M_UR50D. We also train transcoders (an SAE variant) on protein-level representations.

2/ Protein-level representations from PLMs are used in many downstream tasks. Disentangling their features can enhance interpretability, helping us trust and explain downstream applications.

1/ PLMs like ESM have made big strides in predicting protein structure & function. But they feel like a “black-box.” What biological information do PLM representations contain? Can we disentangle them systematically?.

Excited to share our recent work: Sparse autoencoders uncover biologically interpretable features in protein language model representations now in PNAS. Thread below 🧵

RT @TravisEGibson: Congrats Youn! and team, . @lab_berger @HultgrenLab @broadinstitute @mit @GeorgKGerber1 @BrighamWomens @MGBResearchNews….

A new approach to decipher the links between genomic proximity, chromatin conformation, and gene function. We leverage scGPT to derive better quantitative estimates and hypothesize a synergy between TADs and condensates. Joint w @rohitsingh8080 & @hliang74.

Note: our lab has also made a Bluesky! Follow here: For now, lab members will post on both X and bsky.

6/.👥 Contributors: @VUllanat, Bowen Jing, @samsledzieski, and Bonnie Berger.🔗 Code: 📄 Preprint:

5/ Potential applications of MINT. Finally, we show how users can use MINT through two case studies:.MINT predictions align with 23/24 experimentally validated oncogenic PPIs impacted by cancer mutations. MINT estimates SARS-CoV-2 antibody cross-neutralization with high accuracy.

4/ Antibody and TCR–Epitope–MHC Modeling. MINT outperforms IgBert & IgT5 in predicting antibody binding affinity and estimating antibody expression. Finetuning MINT beats TITAN, PISTE and other TCR-specific models on:.TCR–Epitope and TCR–Epitope–MHC interaction prediction.

3/ MINT sets new benchmarks!. It outperforms existing PLMs in:.✅ Binary PPI classification.✅ Binding affinity prediction.✅ Mutational impact assessment. Across yeast, human, & complex PPIs, we see up to 29% gains vs. baselines!

2/ MINT is built on ESM-2 but adds a cross-chain attention mechanism to preserve inter-sequence relationships. We trained MINT on 96 million high-quality PPIs (from STRING-db). Instead of masked language modeling on single sequences, we now capture interaction-specific signals.

1/ Traditional PLMs struggle with PPIs since they model proteins independently. Previous approaches concatenated embeddings or sequences—leading to lost inter-residue context. We fix this with MINT, which allows multiple interacting sequences as input.

We introduce 🌿 MINT (Multimer Interaction Transformer) – a Protein Language Model (PLM) trained on 96M protein-protein interactions (PPIs) to predict binding affinity, mutational impacts, & antibody interactions better than existing PLMs. 🔗Code: 🧵👇.