Great work led by Qihao Liu during his internship with our team @xi_yin_, @Andrew__Brown__, and @YuilleAlan . And props to Qihao for a github release ( with a complete reproduction + releasing models trained on public datasets.

In fact, we find that this simple design scales *even better* than conventional FM with both model size and training steps. Lots of other details, like enabling CFG, the importance of a Variational Encoders in the paper (.

All this is accomplished by a simple modeling pipeline: we show that the input distribution needs to be regularized, and apply a Variational Encoder to the text embeddings. This latent is then evolved via FM to an image latent via a vanilla transformer with *no cross-attention*.

This is cool, but does it work well? Yes! .We show that our approach works well and is par with / outperforms SOTA on a variety of tasks, specifically:.1. Text-to-image.2. Image-to-depth.3. Image-to-text.4. Image super-res

This enables some cool applications, like latent arithmetic:. 1. latent("A white dog wearing a black hat") . latent("Sunglasses") .-latent("A hat").--> An image of a dog with sunglasses without the hat. 2. Interpolate between two text latents --> the image changes smoothly!

Flow matching can transform one distribution to another. So why do text-to-image models map noise to images instead of directly mapping text to images? .Wouldn't it be cool to directly connect modalities together? CrossFlow accomplishes exactly that!

RT @AIatMeta: As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation bench….

Finally @_rohitgirdhar_ and I can talk about our detour into Llama 3 video understanding. You need to understand videos (and caption them 💬) to generate good quality videos! 🐨

Check out Movie Gen 🎥 Our latest media generation models for video generation, editing, and personalization, with audio generation!.16 second 1080p videos generated through a simple Llama-style 30B transformer. Demo + detailed 92 page technical report 📝⬇️.

Llama 3.1 is out! Through adapters we've made it multimodal, supporting images, videos, speech!. Was a fun journey adding video understanding capabilities with @_rohitgirdhar_, @filipradenovic., @imisra_ and the whole MM team! . P.S. MM models are WIP.(not part of the release).

Feels great to have been recognized as an outstanding reviewer for CVPR this year 🤩.

Also check out animate in Meta AI, which builds on top of our prior research on Emu Video and brings your pictures to life! 📹.

Llama 3 is out! Super fortunate to be part of the incredible effort! Also can't wait to share some of our work which is still in progress! 🦙🦙🦙.

Now added ViT-6.5B MAE + MAWS models, and ImageNet-1k finetuned checkpoints. Strong foundational models at all sizes: <100M to >6.5B params. Best public perf on ImageNet-1k linear/finetuned, ImageNetv2, ImageNet-ReaL, low-shot ImageNet-1k & iNaturalist-18.

RT @_rohitgirdhar_: Excited to share what we've been up to this year: Emu Video! A SOTA video generation system from text or images. In spi….

Work done with an incredible team @_rohitgirdhar_ , @Andrew__Brown__ , @quduval , @smnh_azadi , @rssaketh , Akbar Shah, @xi_yin_ , @deviparikh , @imisra_ !!!.

Animating the New York city skyline with "The sun sets and the moon rises"

"A polar bear swimming energetically underwater to catch fish, photorealistic"

Animating a picture of the moon landing with "The American flag waving during the moon landing with the camera panning"

"There's a dog with a harness on that is running through an open field and flying a kite."