Writing

This post is a about diffusion models that have received quite a bit attention recently and such as with DALLE [1] or IMAGEN[2] in a text to image use-case. For me the motivation was to rather understand the underlying mechanics of the method that is surprisingly enough intuitive, but has some non-zero mathematics requirement to be “rigorous” about it. The main paper that motivates this blog post is the (Denoising Diffusion Probabilistic Models)[https://arxiv.org/abs/2006.11239] paper from Ho et al.(2020). As of this moment, the paper is quite recent so I dare to say that this is a very novel method to learning generative models.

This is one of the most typical approaches to reducing variance of MC estimates. Consider the setup where we want to estimate $\E[f(x)]$ with MC, imagine that we have access to a function $h(x)$, now we can compute the expectation of the difference $\E[f(x) - h(x)]$. This might not be that interesting, since by linearity of expectation we would arrive to just $\E[f(x)] - \E[h(x)]$, but things get interesting when we estimate the quantity via MC

First we start with the question, what does it mean for data to be out of distribution. In tradition statistical inference we concern ourselves with the generalization gap. This is defined by the difference in expected error of our algorithm in the training set vs. the test set. Now, in classical approaches the test set is assumed to be sampled from the same distribution, that is the true data distribution \(p(x)\).