Catch us at poster #242 on Thursday at 4:30 PM! More details here: 🔗 Website: https://t.co/IvqpqWVFX9 📄 arXiv: https://t.co/aKu3MSsxhg w/ @ren_wang1 , Bernie Huang, @andrewhowens , and @GopalaSpeech .

And it’s not just speech -- our model can generalize to non-speech sounds, like a barking dog, train chimes, etc.

Our model is trained entirely on unlabeled, in-the-wild videos. Using latent diffusion, it learns to infer the acoustic and ambient properties of different scenes and apply them to input speech.

Unlike methods focusing on room acoustics, our model captures both acoustic and ambient sounds to stylize speech in various environments.

We train a model using self-supervision, leveraging natural videos with recurring sound events and textures. It extracts audio from a video, applies speech enhancement, and then learns to transfer sound properties from one scene to another.

Excited to share our latest work on audio-visual soundscape stylization at #ECCV2024! In this work, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene.

Andrew is a great PhD advisor! I'm pretty sure that you'll learn a lot from his insights and guidance while working together.