When the Score Does Not Point Straight to the Mean

The Question

Why can covariance structure bend or rotate score directions?

Short answer: the score always points in the direction of higher density. In a multivariate setting, unequal variances and correlations can introduce surprises that the score needs to account for. Instead of just pointing along the raw displacement back to the mean, the score is shaped by the covariance structure.

Why I Care

In a previous post, we started to understand the score function in a simplified 1-D setting. Real diffusion models work in many dimensions, so it is essential to analyze the changes that multidimensional distributions bring for someone who wants to become more fluent in modern generative models.

Setup and Definitions

Let's work with a bivariate normal distribution, but write it in vector notation so the geometry is cleaner:

x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}, \qquad \mu = \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix}, \qquad x \sim \mathcal{N}(\mu, \Sigma)

For a two-dimensional Gaussian, the covariance matrix can be written as:

\Sigma = \begin{bmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{bmatrix}

I want to highlight three different cases and their impact on the score function.

Isotropic covariance. This is the nicest case: \(\rho = 0\) and \(\sigma_1 = \sigma_2 = \sigma\). The concrete image is a bullseye. If you throw a dart, any left-right or up-down miss is equally likely. The highest-density target is the mean, and the score points straight back toward it.

Diagonal covariance. Here \(\rho = 0\), but \(\sigma_1 \ne \sigma_2\). For example, a machine might cut plates very precisely in length but not so well in width. The score will be pulled differently by each axis, as we will see.

Correlated covariance. This is the most common case: \(\rho \ne 0\), and often \(\sigma_1 \ne \sigma_2\). A useful example is height and weight. Both coordinates can be marginally normal, but they are positively correlated: usually taller people weigh more. The score still guides us toward higher density, but not necessarily straight to the mean, because locally another direction may correct the anomaly faster. For a very tall person with low weight, the score could point in a direction that lowers height and raises weight.

Let's get our hands dirty with some formulas now.

Minimal Derivation

For finding the score function in a multivariate Gaussian distribution, we'll follow a similar approach as we did for the 1-D case:

We start from the PDF.
We take the log.
We differentiate.

The multivariate Gaussian density is given by:

p(x) = \frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}} \exp\left( -\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) \right)

Taking logs gives us:

\log p(x) = -\frac{d}{2}\log(2\pi) -\frac{1}{2}\log|\Sigma| -\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)

Let's define the precision matrix as:

A = \Sigma^{-1}

and the displacement vector as:

z = x - \mu

So the third term of our log PDF becomes:

-\frac{1}{2}z^T A z

The quadratic form \(z^T A z\) is the squared Mahalanobis distance: distance measured in units of the distribution's own spread, which is exactly the "surprise" from the examples above.

Notice that the first two terms of the log PDF do not depend on \(x\), so they become zero when differentiating with respect to \(x\).

Let's also recall that, because \(A\) is symmetric:

\nabla_z(z^T A z) = 2Az

So we can now derive the score function \(s(x)\):

\begin{aligned} \nabla_x \log p(x) &= \nabla_x \left[ -\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) \right] \\ &= -\frac{1}{2}\nabla_x(z^T A z) \\ &= -\frac{1}{2}(2Az) \\ &= -Az \\ &= -\Sigma^{-1}(x-\mu) \end{aligned}

Therefore:

s(x) = -\Sigma^{-1}(x-\mu)

Let's now see the three cases mentioned above.

Isotropic.

\Sigma = \sigma^2 I = \begin{bmatrix} \sigma^2 & 0 \\ 0 & \sigma^2 \end{bmatrix}

s(x) = -(\sigma^2 I)^{-1}(x-\mu) = -\frac{1}{\sigma^2}(x-\mu)

The score here is similar to the 1-D case: it points straight back toward the mean because both axes are scaled equally.

Diagonal.

\Sigma = \begin{bmatrix} \sigma_1^2 & 0 \\ 0 & \sigma_2^2 \end{bmatrix}

s(x) = - \begin{bmatrix} 1/\sigma_1^2 & 0 \\ 0 & 1/\sigma_2^2 \end{bmatrix} \begin{bmatrix} x_1-\mu_1 \\ x_2-\mu_2 \end{bmatrix} = \begin{bmatrix} -(x_1-\mu_1)/\sigma_1^2 \\ -(x_2-\mu_2)/\sigma_2^2 \end{bmatrix}

The coordinate with smaller variance gets the stronger pull because its precision \(1/\sigma_i^2\) is larger. In the plate example, if the length error is rarer, then that axis is pulled back more strongly for the same displacement from the mean.

Correlated.

\Sigma = \begin{bmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{bmatrix}

s(x) = -\Sigma^{-1}(x-\mu)

The precision matrix \(\Sigma^{-1}\) is a linear transformation of the displacement vector \((x-\mu)\). It can stretch different directions by different amounts and mix the coordinates. In the height/weight example, the score points toward the direction that corrects the anomaly faster, not just toward the mean by drawing a straight line.

For a single Gaussian, the destination does not change: if we keep following the score, we still reach the mean. What changes is the route.

Visualizing the Geometry

Deriving the score gives us the formula, but visualizing its geometry makes the result easier to internalize. In the playground below, we can move the starting point and compare the isotropic, diagonal, and correlated cases to see how covariance changes the direction of the score. For simplicity, the playground fixes the mean at \(\mu=(0,0)\), so the plotted coordinates represent displacements from the mean.

What I Think I Learned

When working in multiple dimensions, the score function takes into account how dimensions affect each other. At first, we seem to lose the intuition we built in the 1-D case. But after thinking about concrete, real-world cases, things become clearer. The math then adds the missing piece: the precision matrix tells us which parts of the displacement are actually surprising.

The key object is the precision matrix:

\Sigma^{-1}

It decides how the raw displacement vector \(x-\mu\) should be transformed before becoming the score. If the covariance is isotropic, the transformation is just a common scale and the score points straight back to the mean. If the covariance has different variances or correlations, the score can be bent away from that raw direction because the distribution's geometry is different.

Open Questions

If we want to go to the peaks of the mountain, how could we do it faster than just following the score?
How do diffusion models learn the score function?

References

Week 5 2-D Gaussian score note
Week 5 2-D Gaussian score field notebook
Bishop, Pattern Recognition and Machine Learning, section 2.3—the multivariate Gaussian and its precision matrix in depth.
Petersen & Pedersen, The Matrix Cookbook—reference for vector-derivative identities like \(\nabla_z(z^T A z) = 2Az\).