Stay tuned to see how our Agents stack up against the latest base LLMs on biomedical question-answering tasks!

👀 Our new Knowledge Agents make https://t.co/fknNWdWJ4k more powerful than ever. 🧬 Get insights backed by the journals and databases that biomedical researchers use daily. 📄 Read the blog more more info:

We continue to evaluate these new models on our benchmark, CARDBiomedBench. Despite significant progress, there are still no models that balance response accuracy and safety on biomedical questions 👀

Check out the latest models in @BiomedArena for all of your biomedical research questions!

Super proud of all the hard work from our team including @tanaynayak99, @owenbianchi_, Shayan Shahand, @DanielKhashabi, and @FarazFaghri!!!

🚨BiomedArena is live🚨 In a partnership with @lmarena_ai, our team at @DataTecnica has released a feedback-rich platform to evaluate LLM performance on real-world biomedical questions. ⚔️Access the arena: https://t.co/LjPiINtIfi 📄Read the blog post:

🚨New LLM benchmark🚨 We're releasing BiomedSQL🔬 for tabular reasoning over large-scale biomedical databases. This includes questions based on implicit scientific conventions—like statistical thresholds, effect direction, and drug approval status. 📄 Preprint:

📄Read the preprint: https://t.co/hl9BPIxT5a 📊Dataset: https://t.co/y6MPciJSwb 💻Code: https://t.co/Qlbl33E0RL Thanks to my teammates at NIH/CARD and @DataTecnica including Maya Willey, Adi Asija, @owenbianchi, Chelsea Alvarado, @mike_nalls, @DanielKhashabi, and @FarazFaghri

We believe this benchmark is a critical step towards building trustworthy text-to-SQL systems that can increase efficiency of lookups for PIs and SMEs, democratize access to biomedical knowledge, and accelerate discovery

Top performers: 1) GPT-o3-mini is the top frontier model with an accuracy of 59% on BiomedSQL, significantly trailing expert-level performance (90%) 2) Our custom text-to-SQL system, BMSQL, improves execution accuracy by ~3% and response quality by ~7% over “vanilla” GPT-o3-mini

Biomedical researchers increasingly rely on large-scale databases that store electronic health records, population-scale studies, and clinical trial information. Our team wanted to test LLMs ability to generate valid SQL queries from questions that domain experts routinely ask

Can LLMs perform reliably as biomedical data analysts? TL;DR: We created the first benchmark designed to challenge LLMs ability to apply scientific reasoning in text-to-SQL generation over biomedical databases, revealing a 30-40% gap between SOTA models and expert performance

Long-form inputs (e.g., needle-in-haystack setups) are the crucial aspect of high-impact LLM applications. While previous studies have flagged issues like positional bias and distracting documents, they've missed a crucial element: the size of the gold/relevant context. In our

Great work from many of our teammates! Let's accelerate data harmonization!

GenoTools: An Open-Source Python Package for Efficient Genotype Data Quality Control and Analysis https://t.co/AkpxXvHFFn #biorxiv_genomic

Koretsky et al. use genome-wide data to cluster patients based on genetic status across risk variants for five neurodegenerative disorders. The results suggest that neurodegenerative diseases have more overlapping genetic aetiology than previously assumed. https://t.co/TD3fv3mdFx

Genetic risk factor clustering within and across neurodegenerative conditions (such as #Parkinsons, #Alzheimers, #ALS) - from @HamptonLLeonard & colleagues; "Neurodegenerative diseases have more overlapping genetic etiology than previously expected" https://t.co/BXdQ4FUCKi

Been a great day for publications from the team. @mkoretsky1 and the team great job on showing genetic overlap and in some instances a depletion of genetic risk across and within neurodegenerative diseases.