🎓 Looking for PhD students, postdocs & interns!.I’m recruiting for my new lab at @NUSComputing, focusing on generative modeling, reasoning, and tractable inference. 💡 Interested? Learn more here: 🗓️ PhD application deadline: June 15, 2025.

RT @danielmisrael: “That’s one small [MASK] for [MASK], a giant [MASK] for mankind.” – [MASK] Armstrong. Can autoregressive models predict….

RT @Vinh_Suhi: 🚀 Exciting news! Our paper "Learning to Discretize Diffusion ODEs" has been accepted as an Oral at #ICLR2025! 🎉. [1/n].We pr….

RT @zhezeng0908: 📢 I’m recruiting PhD students @CS_UVA for Fall 2025!.🎯 Neurosymbolic AI, probabilistic ML, trustworthiness, AI for science….

RT @StephanMandt: My group will be hiring several PhD students next year. I can’t reveal the details before the official announcement at Ne….

RT @gengala13: I’ll be attending #NeurIPS2024, where I’ll present our spotlight: Scaling Continuous Latent Variable Models as Probabilistic….

RT @EmilevanKrieken: @MinkaiX Very cool work! It sounds quite similar to Discrete Copula Diffusion by @liu_anji, in that it combines both p….

Shout-out to my amazing collaborators @OliverBroadrick @Mniepert @guyvdb !!.

[9/n] In addition to presenting an effective discrete diffusion generation algorithm, we emphasize the importance of modeling inter-variable dependencies in discrete diffusion. Please check the details in our paper

[8/n] In particular, by combining a recent discrete diffusion model ( with GPT-2 (small) as the copula model, we achieve better (un)conditioned generation performance in 8-32 fewer denoising steps.

[7/n] Since discrete diffusion models learn all univariate marginals well, we use a copula model to additionally capture the dependency information. As shown on the right side of the figure, by combining information from both models, we generate better samples in fewer steps.

[6/n] Given the prompt: “Let’s do outdoor sports! How about <X> <Y>?” The copula indicates the phrases “alpine skiing” and “scuba diving” happen more often than the others (e.g., “alpine diving”), while the univariate marginals indicate which phrase appears more often.

[5/n] How to fix this problem? We resort to the classical work of Sklar ( that every distribution can be decomposed into two disjoint sets of information: (i) a set of univariate marginals and (ii) a copula.

[4/n] We quantify the performance drop by showing that there is an irreducible component in the negative ELBO stemming from this independent denoising assumption, which prevents the model from generating high-quality samples in just a few steps even if the model is perfect.

[3/n] As shown in the left side of the figure, discrete diffusion models predict the token probabilities individually for each variable. When multiple “edits” are made in a single denoising step, the model fails to consider the probability of these changes happening jointly.

[2/n] This problem can be mitigated by supplementing the missing dependency information with another *copula model* (a deep generative model). Check out details about this problem and our solution in our paper Discrete Copula Diffusion

[1/n] 🚀Diffusion models for discrete data excel at modeling text, but they need hundreds to thousands of diffusion steps to perform well. We show that this is caused by the fact that discrete diffusion models predict each output token *independently* at each denoising step.

RT @PoorvaGarg11: Are you looking for an inference algorithm that supports your discrete-continuous probabilistic program? Look no further!….

RT @KareemYousrii: I'm very happy to share PyJuice, the culmination of years of scaling up probabilistic & logical reasoning for neural net….

Interested in the technical details of PyJuice? Our paper has been accepted to #ICML24 and is available on Arxiv Many thanks to my amazing collaborators @KareemYousrii and @guyvdb, without whom this project would not be possible.