Expected Value of a Simple Forward Process

The Question

Why does adding Gaussian noise to the same image many times leave it unchanged on average?

What does that even mean?

This does not mean that one noisy image is unchanged. It means the distribution of noisy images is centered around the original image.

Short answer: intuitively, the noise added in multiple experiments will cancel out with the noise from the other experiments, but let's unpack this.

Why I Care

One of the building blocks of diffusion models is the so-called forward process, in which we corrupt "clean" data, like images, by adding Gaussian noise. The goal here is to learn how to undo the corruption process, and hence generate images from noise. Brilliant!

I think these kinds of concepts really click once you have both the right intuition and the math/derivation understanding. They work in tandem, and usually one helps the other.

Taking this into account, let's see one of the properties, the expected value, of a simplified forward process.

Setup and Definitions

For concreteness, imagine that you take an image of your cat, \(x_0\). Then you generate random Gaussian noise with the same shape as your image, \(\varepsilon \sim \mathcal{N}(0, I)\). Finally, you take your original image and add the noise, scaled by a factor \(\sigma\).

Here \(\sigma\) is the noise level: how strong you want the noise to be. For example, if a pixel's noise value is \(1.3\), but \(\sigma = 0.1\), your final pixel increases by \(0.13\). Then you save the output of this operation in \(x_t\).

Side note: for simplicity, let's not worry about image clipping or any other practical consideration that appears in a real-world scenario.

Congratulations, you just implemented a toy version of a forward diffusion process:

x_t = x_0 + \sigma \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)

Equivalently, conditioned on the original image, the noisy image is distributed as:

x_t \mid x_0 \sim \mathcal{N}(x_0, \sigma^2 I)

So \(x_0\) is the center of the noisy-image distribution.

Now imagine you run this experiment a bunch of times. You'll get many different noisy images:

x_t^{(1)}, x_t^{(2)}, \ldots, x_t^{(M)}

What would you imagine happens if you take the average of your \(x_t\)'s?

\frac{1}{M}\sum_{m=1}^{M} x_t^{(m)}

By the law of large numbers, as \(M\) grows, this average converges to the clean image \(x_0\).

\frac{1}{M}\sum_{m=1}^{M} x_t^{(m)} = x_0 + \sigma \frac{1}{M}\sum_{m=1}^{M}\varepsilon^{(m)}

Since the average noise term goes to zero, the average noisy image goes back to \(x_0\).

Minimal Derivation

Forward process:

x_t = x_0 + \sigma \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)

\(x_0\): original point or image.
\(\sigma\): noise level.
\(\varepsilon \sim \mathcal{N}(0, I)\): Gaussian noise independent from \(x_0\).

We know \(x_0\), and want to find:

\mathbb{E}[x_t \mid x_0]

\begin{aligned} \mathbb{E}[x_t \mid x_0] &= \mathbb{E}[x_0 + \sigma \varepsilon \mid x_0] \\ &= \mathbb{E}[x_0 \mid x_0] + \mathbb{E}[\sigma \varepsilon \mid x_0] \qquad \text{by linearity} \\ &= x_0 + \mathbb{E}[\sigma \varepsilon \mid x_0] \qquad \text{since } x_0 \text{ is known} \\ &= x_0 + \sigma \mathbb{E}[\varepsilon \mid x_0] \qquad \text{since } \sigma \text{ is deterministic} \end{aligned}

Since \(\varepsilon\) is independent from \(x_0\):

\mathbb{E}[\varepsilon \mid x_0] = \mathbb{E}[\varepsilon]

And since \(\varepsilon \sim \mathcal{N}(0, I)\):

\mathbb{E}[\varepsilon] = 0

Therefore:

\mathbb{E}[x_t \mid x_0] = x_0 + \sigma \cdot 0 = x_0

What I Think I Learned

The expected value of a forward process, as defined in this post, is the initial point itself.

This does not mean that a single noisy image is unchanged. It means that if we add fresh zero-mean Gaussian noise to the same image many times, the noisy images are centered around the original image.

Open Questions

What happens with the standard deviation?
How does normalizing or clipping affect the results?
How does this change in the standard DDPM forward process, where \(x_0\) is also scaled down?

References

Week 2 forward moments notebook
Week 2 expectation toolkit
Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015)—introduced diffusion probabilistic models built around an iterative forward diffusion process.
Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020)—the standard DDPM forward process from the Open Questions, where \(x_0\) is also scaled down.