Excited to share that our paper "Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder" is accepted to #ACL2025! .Preprint: Thank @SimonShaoleiDu and @PangWeiKoh so much for your support and guidance throughout the journey!.

RT @thao_nguyen26: Web data, the “fossil fuel of AI”, is being exhausted. What’s next?🤔.We propose Recycling the Web to break the data wall….

RT @avibose22: 🧠 Your LLM should model how you think, not reduce you to preassigned traits.📢 Introducing LoRe: a low-rank reward modeling f….

RT @RulinShao: 🎉Our Spurious Rewards is available on ArXiv! We added experiments on.- More prompts/steps/models/analysis. - Spurious Prom….

RT @jcqln_h: LMs often output answers that sound right but aren’t supported by input context. This is intrinsic hallucination: the generati….

RT @StellaLisy: 🤯 We cracked RLVR with. Random Rewards?!.Training Qwen2.5-Math-7B with our Spurious Rewards improved MATH-500 by:.- Rando….

Inspiration for future research: (1) Effectively utilizing vision encoders offers benefits without pretraining new vision models. (2) Promptable image embeddings boost performance on fine-grained tasks. Feel free to contact me if you are interested in our work!.

Finding 7: The question in the prompt greatly helps the extraction and utilization of visual information. We remove the question in the prompt for the evaluation of LLaVA-1.5-VLM2Vec, and the performance drops to CLIP's level.

Finding 6: Text generation is not the only solution to fine-grained visual reasoning. We convert the LLaVA-1.5 model to a CLIP-like contrastive VLM using VLM2Vec. Surprisingly, it still performs well on What'sUp!.

Finding 5: A stronger text encoder does not suffice towards solving the task. We replace the CLIP text encoder by the text encoder in LLM2CLIP, but the accuracy is still low.

Finding 4: Using multiple text tokens similarly does not help. We use all text tokens of CLIP text encoder, but observe no improvement.

Finding 3: Part of the information comes from the order of patch tokens. If we use all patch tokens and their order using RoPE when finetuning CLIP, the performance improves!.

Finding 2: Detailed information resides in the patch tokens. If trained with only the [CLS] token of CLIP vision encoder, LLaVA-1.5's performance will drop a lot.

Finding 1: Training data alone does not lead to stronger extraction ability. We finetune the CLIP using converted LLaVA-1.5's data, but found no big improvement. Thus, training data alone does not help.

This exploration originates from our observation that Generative MLLMs using the same, fixed CLIP vision encoder and weights achieve significantly higher accuracy than CLIP in many visual reasoning benchmarks like Winoground, NaturalBench, MMVP, and What'sUp. Then we conducted a.

We found that the CLIP vision encoder is more powerful than you might think, if you extract and utilize its embedded visual information more effectively, e.g., using a Generative MLLM as the extractor! . Wondering how and why? Check the details below: