Can visual SSL match CLIP on VQA?. Yes! We show with controlled experiments that visual SSL can be competitive even on OCR/Chart VQA, as demonstrated by our new Web-SSL model family (1B-7B params) which is trained purely on web images – without any language supervision.

RT @hou_char: Come by the poster if you want recommendations on cool restaurants to try in Vancouver 😃!.

Congrats @jianyuan_wang and co!!!.

Welcome Rob! So blessed to have you steer the ship! See you around the office :).

RT @cindy_x_wu: Introducing COMPACT: COMPositional Atomic-to-complex Visual Capability Tuning, a data-efficient approach to improve multimo….

Web-SSL model weights are now available on GitHub and HuggingFace! You may use your favorite Transformers library API calls or load the model with native PyTorch - up to your preference. For more usage details, please see HuggingFace collection:.

Excited to release the training code for MetaMorph! MetaMorph offers a simple yet effective way to convert LLMs into a multimodal LLM that not only takes multimodal inputs, but also generates multimodal outputs via AR prediction. This confers the ability to “think visually”, and.

RT @_amirbar: Excited to share that our paper on Navigation World Models was selected for an Oral presentation at CVPR!.Code & models:. htt….

[8/8] For more details, please check out the paper and website. Paper: Website: It was a great privilege and so much fun to work with @TongPetersb, @JiachenAI, @koustuvsinha, @liuzhuang1234, @endernewton, Michael Rabbat, Nicolas.

[7/8] This side project started in October when @TongPetersb, @_amirbar, and I were thinking about the rise of CLIP as a popular vision encoder for MLLMs. The community often assumes that language supervision is the primary reason for CLIP's strong performance. However, we.

[6/8] Developing higher resolution SSL models is a promising direction. High resolution versions of Web-DINO 7B can nearly match the performance of SOTA off-shelf CLIP-family models such as SigLIP 2, despite seeing only on images.

[5/8] The observed scaling behavior is not unique to DINOv2; other visual SSL methods such as MAE also show similar potential!

[4/8] Web-SSL also performs well on classic vision tasks and is competitive with the original DINOv2. This means we are getting closer to developing vision encoders that excel at both pure vision and multimodal capabilities!

[3/8] Why can SSL learn text-sensitive features purely from images?.Hypothesis: Web images contain text. Observation: Training Web-DINO on text-heavy image subsets drastically improves OCR/Chart VQA performance, matching CLIP trained on the full data.

[2/8] Web-DINO (SSL) scales better with model size from 1B - 7B params than CLIP, especially on OCR/Chart VQA – proving that vision-only models can also perform well on tasks that were traditionally dominated by CLIP.

[1/8] We trained SSL (Web-DINO) and CLIP on the same web-scale dataset (MC-2B images/image-text pairs) and evaluated fairly on VQA by freezing the vision encoders. Our apples-to-apples comparisons enable novel insights into the scaling behavior of SSL in this new data regime.

RT @liuzhuang1234: New paper - . Transformers, but without normalization layers (1/n)

@TongPetersb @JiachenAI @YoungXiong1 @endernewton @koustuvsinha @ylecun @sainingxie @liuzhuang1234 Update: @askalphaxiv generously decided to feature MetaMorph! We will be answering questions for a week, so feel free to comment on: