✨New preprint: Dual-Process Image Generation! We distill *feedback from a VLM* into *feed-forward image generation*, at inference time. The result is flexible control: parameterize tasks as multimodal inputs, visually inspect the images with the VLM, and update the generator.🧵

If you're at #ICCV2025, swing by our poster today! We'll show how you can directly fine-tune an image generator with a VLM's language modeling loss. 📍 https://t.co/bu6WrGtVrF 🔗 https://t.co/67P9uxWS1y cc @jongranskog @holynski_ @trevordarrell

Check out our new project at https://t.co/ApCyLKjo4F! Go and poke the visualizations on our website; you might even find a few easter eggs 🐣

I'm presenting a poster at #ICML2025 today! Stop by if you want to learn how VLMs encode different representations of the same task (spoiler: it's the same). 🌐 https://t.co/Dm5PqGefkW 🔗 https://t.co/TCsFZ21Npa cc @_amirbar @trevordarrell

(6/n) Special thanks to my co-authors @jongranskog, @holynski_, @trevordarrell! This was a great academic collaboration with @runwayml, especially @agermanidis, who was super supportive of our very experimental ideas throughout the entire process.

(5/n) Research is non-linear! We started this project more than a year ago – first by training a diffusion captioner (a VLM that encodes images with diffusion hyperfeatures). We’re not working on that direction anymore, but here’s a peek at that first prototype:

(4/n) Check out our paper + code! Our codebase lets you mix-and-match different off-the-shelf image generators and VLMs, and can run on Nvidia RTX 4090s. Page: https://t.co/67P9uxWS1y Paper: https://t.co/w5LRDO7lWg Code:

(3/n) You can get pretty creative, because VLMs afford a flexible interface. We play around with implementing spatial controls via *visual prompting*, where we overlay the control over the image and ask the VLM if they match, then optimize for an image that matches the control.

(2/n) Our method re-uses the visual instruction tuning loss originally used to train the VLM, to instead optimize the weights of the image generator.

(7/n) Check out our paper and website for more info! Website: https://t.co/b4mq6B1PlX Paper: https://t.co/hqbPcuCgDl Code:

(6/n) The idea of a task vector is not new; we study its cross-modal properties following prior work in LLMs (e.g., https://t.co/gACnnwkCyI, https://t.co/fRIO6s3ZT4).

(5/n) The most surprising result, at least to me, is that task vectors can be patched from the base LLM to its corresponding fine-tuned VLM. Here we patch from Mistral to Idefics2. This means that the VLM can re-purpose functions learned entirely in language onto image queries.

(4/n) Motivated by this similarity in task representations, we explore mixing and matching the task specification and query format. We call this cross-modal patching.

(3/n) We start by looking at how the model processes the final token to execute the task. Conditioned on either text or image ICL, the token undergoes three phases across model layers: it first resembles the colon input, a meta-summary of the task, then the final answer.

(2/n) Our main finding is that task representations in VLMs are consistent across modality (text, image) and specification (example, instruction).

In a new preprint, we show that VLMs can perform cross-modal tasks... ...since text ICL 📚, instructions 📋, and image ICL 🖼️ are compressed into similar task representations. See “Task Vectors are Cross-Modal”, work w/ @trevordarrell, @_amirbar. https://t.co/b4mq6B1PlX

Our Knowledge in Generative Models workshop #ECCV2024 is happening in a few hours! ⏰ Monday, Sept 30th, 2-6PM CEST 📍 Brown 2 (note location change from Brown 1) 🔗 https://t.co/3BWyDHaFNn

We are organizing a new workshop on "Knowledge in Generative Models" at #ECCV2024 to explore how generative models learn representations of the visual world and how we can use them for downstream applications. For the schedule and more details, visit our website: 🔗Website:

Come drop by our poster for🔮Readout Guidance at #CVPR2024 this Wednesday! 📍Arch 4A-E, Poster #332 📅Wed 19 June 5-6:30PM PST 🌐 https://t.co/GyV2tghSJk 🔗 https://t.co/EZSQeo2EeR w/ @trevordarrell, @oliver_wang2, @danbgoldman, @holynski_

Update on 🔮Readout Guidance ( https://t.co/EZSQeo26pj)! We open sourced the code – check out our demos, model weights, and training code: https://t.co/oNSM9IeA6B. Here’s a teaser of what you can do with our method: