Adobe's #GenerateSoundtrack is LIVE today! 🎉 Studio-quality music for storytellers🎵 * Trained on #Licensed data, * Commercially safe, royalty-free, and cleared for any use, & * Exported with #ContentCredentials for transparency and attribution. Get started:

Thrilled to announce “MIDI-LLM: Adapting LLMs for Text-to-MIDI Music Generation” w/ @huangcza and Yoon Kim! 🎸 Live Demo https://t.co/6N6AqyrZuW 💻  https://t.co/Z1v82uool5 🤗 https://t.co/k9deMY82Vk From a text prompt, it generates MIDIs you can edit directly in a DAW 🧵

Congrats @JCJesseLai and Team!

Adobe’s new AI audio tools can add soundtracks and voice-overs to videos

Adobe announced DRAGON on Hugging Face Distributional Rewards Optimize Diffusion Generative Models

DRAGON introduces a new approach to designing and optimizing reward functions to enhance human-perceived quality.

With an appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality win rate without training on human preference annotations.

Over all 20 target rewards, DRAGON achieves an 81.45% average win rate. Moreover, reward functions based on exemplar sets indeed enhance generations and are comparable to model-based rewards.

We further compare instance-wise (per-song) and full-dataset FAD settings while ablating multiple FAD encoders and reference sets.

For evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20 different reward functions, including a custom music aesthetics model, CLAP score, Vendi diversity, and Frechet audio distance (FAD).

Then, DRAGON gathers online and on-policy generations, scores them to construct a positive demonstration set and a negative set, and leverages the contrast between the two sets to maximize the reward.

When cross-modality encoders such as CLAP are used, the reference examples may be of a different modality (e.g., text versus audio).

Leveraging this versatility, we construct novel reward functions by selecting an encoder and a set of reference examples to create an exemplar distribution.

It can optimize reward functions that evaluate either individual examples or distributions of them, making it compatible with a broad spectrum of instance-wise, instance-to-distribution, and distribution-to-distribution rewards.

Compared with traditional reinforcement learning with human feedback (RLHF) or pairwise preference approaches such as direct preference optimization (DPO), DRAGON is more flexible.

Introducing "DRAGON: Distributional Rewards Optimize Diffusion Generative Models"! 📖: https://t.co/biSceK5vgQ 🎹: https://t.co/6taiaC8SUZ A new framework for fine-tuning gen models towards a target distribution. By Yatong Bai w/@CasebeerJonah @somayeh_sojoudi @NicholasJBryan

Hyped to be presenting Presto next week at #ICLR2025 as a Spotlight! Super excited to chat about all things generative music/audio, diffusion acceleration, and real-time audio systems, lmk if you want to meet (DMs open)!

Our paper "Presto! Distilling Steps and Layers for Accelerating Music Generation" has been accepted at #ICLR2025 @iclr_conf! Check out the demo video and links below, excited to see everyone in Singapore 🇸🇬!

🎶Diffusion Inference-time T-Optimization (DITTO) for music gen gets faster w/DITTO2! Optimize diffusion initial noise latent via a 1-step sampler for powerful control. Decode w/few sampling steps for best quality. Led by @zacknovack w/@McAuleyLabUCSD @BergKirkpatrick, myself

Ultra fast music generation via accelerating diffusion TTM without quality loss. Amazing work led by @zacknovack! Summer research intern work at @AdobeResearch w/@__gzhu__, @CasebeerJonah, myself, and University collab w/ @McAuleyLabUCSD @BergKirkpatrick at UCSD!