What can you do when you need to evaluate a set of models but don't have too much labeled data? Solution: add unlabeled data. It was great to co-lead this project with @dmshanmugam and come talk to us at NeurIPS!

@rohitsingh @alexpywu @NatureComms link:

We are thrilled to finally publish @rohitsingh and @alexpywu's model of single-cell causality through the lens of a cell-state parallax! Now out in @NatureComms.

Authors: @gujral_onkar, @mihirbafna14, @ejalm and @lab_berger

8/ Paper: https://t.co/G2TUaVAghn MIT News article: https://t.co/Py9CdEXXmx Github:

7/ We show that SAE & transcoder features are much more interpretable than ESM neurons, for both protein-level & amino acid-level representations. This has the potential to improve safety, trust & explainability of PLMs. As PLMs improve, SAEs could help us learn new biology.

6/ We also use Claude to autointerpret SAE features based on protein names, families, gene names & GO terms. Many features correspond to families (like NAD Kinase, IUNH, PTH) & functions (like methyltransferase activity, olfactory/gustatory perception).

5/ We interpret these SAE features using Gene Ontology (GO) enrichment. Many protein-level SAE features align tightly with GO terms across all levels of the GO hierarchy.

4/ SAEs have a very wide latent dimension with a sparsity constraint. This forces PLM representations to disentangle into biologically interpretable, sparsely activating features without any supervision.

3/ We train sparse autoencoders (SAEs) on protein-level and amino acid-level representations from layers 6-10 of ESM2_t12_35M_UR50D. We also train transcoders (an SAE variant) on protein-level representations.

2/ Protein-level representations from PLMs are used in many downstream tasks. Disentangling their features can enhance interpretability, helping us trust and explain downstream applications.

1/ PLMs like ESM have made big strides in predicting protein structure & function. But they feel like a “black-box.” What biological information do PLM representations contain? Can we disentangle them systematically?

Excited to share our recent work: Sparse autoencoders uncover biologically interpretable features in protein language model representations now in PNAS. Thread below 🧵

Congrats Youn! and team, @lab_berger @HultgrenLab @broadinstitute @mit @GeorgKGerber1 @BrighamWomens @MGBResearchNews @harvardmed @AshleeMEarl

A new approach to decipher the links between genomic proximity, chromatin conformation, and gene function. We leverage scGPT to derive better quantitative estimates and hypothesize a synergy between TADs and condensates. Joint w @rohitsingh8080 & @hliang74

Note: our lab has also made a Bluesky! Follow here: https://t.co/qz6taL98Vb. For now, lab members will post on both X and bsky.

6/ 👥 Contributors: @VUllanat, Bowen Jing, @samsledzieski, and Bonnie Berger 🔗 Code: https://t.co/GqrXMjkBBK 📄 Preprint:

5/ Potential applications of MINT Finally, we show how users can use MINT through two case studies: MINT predictions align with 23/24 experimentally validated oncogenic PPIs impacted by cancer mutations. MINT estimates SARS-CoV-2 antibody cross-neutralization with high accuracy.

4/ Antibody and TCR–Epitope–MHC Modeling MINT outperforms IgBert & IgT5 in predicting antibody binding affinity and estimating antibody expression. Finetuning MINT beats TITAN, PISTE and other TCR-specific models on: TCR–Epitope and TCR–Epitope–MHC interaction prediction.

3/ MINT sets new benchmarks! It outperforms existing PLMs in: ✅ Binary PPI classification ✅ Binding affinity prediction ✅ Mutational impact assessment Across yeast, human, & complex PPIs, we see up to 29% gains vs. baselines!