Check out M3DocRAG -- multimodal RAG for question answering on Multi-Modal & Multi-Page & Multi-Documents (+ a new open-domain benchmark + strong results on 3 benchmarks)! ⚡️Key Highlights: ➡️ M3DocRAG flexibly accommodates various settings: - closed & open-domain document

M3DocRAG consists of 3 stages: 1) Extract visual embedding (e.g., w/ ColPali) from each page image. 2) Retrieve top-K pages (+ approximate indexing for faster search in open-domain setting) 3) Generate an answer with a multimodal LM (e.g., Qwen2-VL) given the retrieved K

In MP-DocVQA (closed-domain DocVQA with up to 20 pages), while the text RAG pipeline falls short compared to existing approaches, all multimodal pipelines outperform their text-based counterpart. Notably, our M3DocRAG delivers the state-of-the-art results.

In M3DocVQA (open-domain DocVQA on 3K docs), our M3DocRAG (ColPali + Qwen2-VL 7B) significantly outperforms text RAG (ColBERT v2 + Llama 3.1 8B), across all different evidence modalities / question hops / # pages. The gap is bigger when the evidence involves images.

In MMLongBench-Doc (closed-domain DocVQA with up to 120 pages), our M3DocRAG (ColPali + Qwen2-VL 7B) achieves the best scores in most settings. This demonstrates the effectiveness of multimodal retrieval over handling many pages by concatenating low-resolution images.

Researchers from Bloomberg and UNC Chapel Hill Introduce M3DocRAG: A Novel Multi-Modal RAG Framework that Flexibly Accommodates Various Document Context Researchers from UNC Chapel Hill and Bloomberg have introduced M3DocRAG, a groundbreaking framework designed to enhance AI’s

M3DocRAG : Multi-modal RAG System This paper presents M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.).

Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.

M3DocRAG, a novel framework that enhances retrieval-augmented generation (RAG) by integrating multi-modal elements, including text, charts, and figures. This approach not only accommodates various document contexts but also improves performance in answering complex questions

M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions

A very interesting paper : M3DocRAG = ColPali on multi-pages & documents + Pages retrieval Bravo to the team for the hard work 👏 From an IR perspective, these scenarios are really interesting imo =

複数文書とページを横断的に理解するマルチモーダルAI対応のRAGフレームワーク「M3DOCRAG」：Bloombergとノースカロライナ大学の研究チーム

🎈 M3DocRAG：マルチモーダルRAGシステム！ ✎. https://t.co/HOJIY1dkZi M3DocRAGは、単一ページから複数ページのドキュメントまで、さまざまな形式のドキュメントに対応できる質問応答システムだそうです！ テキスト、テーブル、画像など、さまざまな要素を扱えるそうです！

https://t.co/7gnmTdsBcF Authors unveil M3DocRAG, an innovative multi-modal framework for answering questions across diverse documents, handling multiple pages and visual data. It outperforms baselines on benchmarks like M3DocVQA, showcasing its impressive capabilities.

The freshest AI/ML researches of the week, part 2 ▪️ WEBRL: Training LLM Web Agents ▪️ DynaSaur: Large Language Agents ▪️ THANOS: Skill-Of-Mind-Infused Agents ▪️ DELIFT ▪️ HtmlRAG ▪️ M3DOCRAG ▪️ Needle Threading ▪️ Survey Of Cultural Awareness In LMs ▪️ OPENCODER ▪️ Polynomial

Exciting breakthrough in Document AI! Researchers from UNC Chapel Hill and @business have developed M3DocRAG, a revolutionary framework for multi-modal document understanding. The innovation lies in its ability to handle complex document scenarios that traditional systems

6. M3DOCRAG: Multi-Modal Retrieval For Document Understanding Introduces a multimodal RAG framework to handle multi-page and document QA tasks with visual data. https://t.co/Fuf2Xu4sRN

Researchers from Bloomberg and UNC Chapel Hill Introduce M3DocRAG: A Novel Multi-Modal RAG Framework that Flexibly Accommodates Various Document Context

Great that Bloomberg is involved in this https://t.co/qwZLHnUoyO

はてなブログに投稿しました ［論文メモ］M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding - 子供の落書き帳 Renaissance https://t.co/RgkVedei3M #はてなブログ