🚨ICML Paper Alert🚨.What if finding the right protein homologs wasn't a slow search, but a learned part of the model itself?.We introduce 𝐏𝐫𝐨𝐭𝐫𝐢𝐞𝐯𝐞𝐫, an end-to-end framework that learns to retrieve the most useful homologs for self-supervised reconstruction! (1/12)

Check out the full paper, @NotinPascal intro blogpost, and follow for future code release at:.paper: code: blogpost: (12/12).

Huge thanks to the team for this great collaboration between @DeboraMarksLab and @OATML_Oxford with @NotinPascal @PeterM_rchGroth @lood_ml @deboramarks @yaringal. (11/12).

We used the powerful architectures PoET from Bepler et Truong as the reader model and ESM2 from @alexrives @ebetica as the Retriever. Thanks for providing such great, easy-to-use open models!.Thanks to the team at @meta, @gizacard for the Atlas model inspiration. (10/12).

Our vector framework is also uniquely flexible, leveraging FAISS indexing. The entire UniRef50 database becomes a single ~13GB file Index. You can easily add new or proprietary sequences to the index just by embedding them—no retraining of the full model required. (9/12).

A crucial detail: the starting point for this training matters. ESM-2's default embeddings are a poor start for retrieval. We found initializing with a pretrained dense passage retriever creates a much better foundation for our final end-to-end learning. (8/12)

The speedup is dramatic. By replacing slow alignment with learned vector search, Protriever is two orders of magnitude faster than MMseqs2-GPU. This efficiency comes with superior predictive accuracy too in our end-to-end framework. (7/12)

This joint training leads to new state-of-the-art results. On the ProteinGym benchmark, Protriever is the top-performing sequence-based model, achieving a Spearman correlation of 0.479 and outperforming baselines across all metrics. (6/12)

The key is our 𝐞𝐧𝐝-𝐭𝐨-𝐞𝐧𝐝 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠. The Reader's performance on its self-supervised task - reconstructing the query protein from its homologs - provides a direct learning signal to the Retriever. This teaches the Retriever which homologs are truly relevant. (5/12).

Protriever contains two core modules: The 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐞𝐫 embeds proteins into a vector space for fast similarity search. The 𝐑𝐞𝐚𝐝𝐞𝐫 then uses these retrieved homologs as context. An 𝐈𝐧𝐝𝐞𝐱 represents all Uniref50 sequences, the reader pulls from. (4/12).

But finding the 𝐫𝐢𝐠𝐡𝐭 homologs is a major bottleneck. Traditional methods are slow & operate independently of downstream models. Inspired by retrieval-augmented NLP models, we propose a unified end-to-end differentiable process. (3/12).

Accurately predicting a protein's fitness landscape is critical for pathogenicity prediction and designing new drugs. The most powerful models rely on homologs - related sequences - to reveal which mutations are permissible, as shown in the ProteinGym benchmark (2/12)

RT @H__Spinner: Marks Lab is HIRING!!! If you're a software engineer interested in biology, proteins, RNAs, viruses, genomes, etc etc and M….

RT @NatureBiotech: The February issue, with a focus on protein engineering, is live Our cover shows the three data….

RT @NotinPascal: 🚨The new version of ProteinGym is out!🚨.#NeurIPS2023 #GenBio #Proteins #Benchmark #MAVE https://t.….

RT @NotinPascal: 📢Very pleased to be presenting our ProteinNPT paper at NeurIPS on Tuesday!📢We introduce a novel semi-supervised conditiona….

RT @roseorenbuch: Announcing popEVE - a deep generative model of the human proteome that reveal over a hundred novel genes involved in rare….