Proud to share that my paper w/ @frankdonaldwood & @sirbayes is a #ICML2025 spotlight!. TLDR: We find a simple patch-based mechanism that closely approximates how image diffusion models generalize, regardless of architecture.

I'll be presenting this today in the evening poster session from 4:30pm to 7. Come say hi if you want to discuss diffusion generalization!.

We’re excited to keep working to improve PSPC and close the gap to network denoisers! We hope PSPC will lay the groundwork for a better understanding of how diffusion models generalize, and allow us to attribute which training samples are responsible for each generated image 11/.

However, there is still work to do! Although PSPC samples have some similarity to network samples, error accumulation over the diffusion process results in artifacts in the final samples. 10/

Comparing PSPC to network denoiser outputs, we find that it closely approximates network outputs across all diffusion times and datasets - suggesting that local denoising comprises a key component of diffusion generalization 9/

Can we combine patch-based denoisers to reproduce diffusion model generalization? We propose compositing patch posterior means with a simple mechanism which we call Patch Set Posterior Composite (PSPC) 8/

If network denoisers use local operations during training, we would expect the same mechanism to apply for sampling. Indeed, we find that patch posterior means are good estimators of network output patches when run on intermediate noisy images drawn from the sampling process. 7/.

Why would diffusion models use local operations? We find that for most of the training distribution, patch posterior means are equivalent to optimal denoiser patches. For much of training, a local estimator is sufficient to estimate the minima of the training objective! (left) 6/

We hypothesize that these gradients are the result of network denoisers employing local denoising operations. We approximate these operations with a patch posterior means - a weighted average of the same patch of each training set image 5/.

Looking at the gradients of neural-network denoisers suggests a local inductive bias. We find networks are more sensitive to pixels within a local patch around the output pixel. The size of this patch increases with diffusion time. 4/

Our work analyzes these approximation errors. We find that in both U-Net and DiTs, neural network denoisers make similar errors in both magnitude and quality. This suggests a shared inductive bias common to all image diffusion models 3/

However, this optimal denoiser has a problem - it can’t generalize. Sampling with the optimal denoiser generates training set copies. The generalization behaviour of diffusion models is therefore the result of neural network approximation errors 2/.

Diffusion models produce samples through repeated denoising operations - they estimate clean images from ones which has been corrupted by Gaussian noise. Notably, the optimal denoising operation has a simple closed form. It is a weighted average of training set images. 1/.