Special thanks to my co-first author Kim-Celine Kahl (@kc_kahl) for the valuable contribution to this project to push research in this field! .Also thanks to the collaborators in this project - @jeremiastraub , @CarTLueth, @lena_maierhein, @maierhein and @pfjaeger.

📊 With our large-scale robustness study we find that: .1. No fine-tuning method consistently outperforms others in robustness.2. Robustness varies more by shift than by method.3. Simple image-free baselines perform surprisingly well and LoRA performs best on in-distribution data.

💡We outline three requirements for robustness evaluation:.1. Measure robustness on real-world shifts, as synthetic shifts may not reflect real-world scenarios.2. Move beyond token-matching metrics, as they often miss semantic nuances.3. Use sanity baselines for interpretability.

🛠️ To address this gap, we:.1. Identify pitfalls and define core requirements for robust VLM evaluation. 2. Introduce SURE-VQA — an open-source framework for systematic robustness evaluation. 3. Validate it empirically and benchmark the robustness of various fine-tuning methods.

❓Vision-Language Models (VLMs) hold promise for medical VQA, but their robustness to unseen data is a concern. Evaluating this requires controlled setups for systematic insights—yet current approaches often fall short, limiting their ability to accurately assess model robustness.

🔍 TLDR: We reveal gaps in current robustness evaluations of medical vision-language models and introduce SURE-VQA, a framework addressing them via real-world shifts, improved metrics, and sanity baselines—validated with a large study across 3 datasets and 4 distribution shifts.

📣 I'm excited to share that our paper "SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks" has been accepted to #TMLR !. 📃 Read our paper: 💻 Check out the code: More ⬇️

RT @CarTLueth: We’re thrilled to welcome Eric Brachmann (@eric_brachmann), Senior Staff Scientist at Niantic Spatial, Inc. and a leading ex….

RT @CarTLueth: In case you missed the last talky by Prof. Yuki Asano (@y_m_asano) on "Post-Pretraining in Vision, a….