Research Log

In Search of PCA’s Lost Variance

September 17, 2026 · PCA · Linear algebra

The Question

In PCA, we compress our data by finding their principal components and then selecting the first \(k\) components with the largest eigenvalues. By doing this, we discard some principal components associated with lower eigenvalues. What do these discarded eigenvectors and their corresponding eigenvalues have to do with the reconstruction mean-squared error?

Short answer: in the setting of this post, the reconstruction MSE equals, in fact, the sum of the discarded eigenvalues.

By MSE, I mean the average squared distance between each sample \(x_i\) and its reconstruction \(\widehat{x}_i\):

\[ \operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}\lVert x_i-\widehat{x}_i\rVert^2. \]

Here, \(n\) is the number of samples: we add the squared errors across coordinates, then average over samples. I’ll also use \(1/n\) to normalize the covariance matrix.

Note: I encourage the reader to check this notebook as a companion to this blog post.

Why I Care

Compressing data is a common theme in AI/ML. For example, some approaches to diffusion models work in the latent space to reduce computation. But compression comes with its own challenges, one of them being aware of what we are losing with these smaller representations (as a side note, I highly recommend the 3Blue1Brown series “Compression is intelligence”).

Today, I’d like to explore what’s gone after we reduce our data dimensions using PCA. In doing that, I hope we can get a nice geometric intuition for some topics related to probability—mean, variance—linear algebra—projections, eigendecomposition—and machine learning, namely PCA.

Setup and Definitions

Covering PCA in detail is out of the scope of this post. Instead, I’ll use a motivating example that sheds light on the question at hand.

Let’s start with these data in 2D.

Synthetic two-dimensional data forming an elongated cloud away from the origin before centering.
Synthetic 2D data before centering, from the notebook.

In PCA, we start by centering the data, such that the empirical mean becomes zero. Writing the centered points as \(\tilde{x}_i=x_i-\bar{x}\),

\[ \frac{1}{n}\sum_{i=1}^{n}\tilde{x}_i=0. \]

This will pay off later in our “Derivation” section.

PCA describes the variation around the empirical mean, which we ensured equals zero in our previous step. To achieve this, PCA applies eigendecomposition to the covariance matrix \(S\):

\[ S=\frac{1}{n}X_c^{\mathsf T}X_c, \]

where \(X_c\) has the centered data points as its rows.

We can visualize the previous steps in the following image.

Centered data with a long red arrow along the first principal direction and a shorter green arrow along the perpendicular second principal direction.
PCA directions from the covariance matrix, from the notebook. The arrows follow the principal directions, with lengths \(2\sqrt{\lambda_j}\), twice the standard deviation along each direction. Reversing an arrow gives the same principal axis.

For our example, we have the following two principal directions, each with length one.

First principal direction:

\[ \text{eigenvector }v_1\approx \begin{bmatrix}-0.86\\-0.51\end{bmatrix}, \qquad \text{eigenvalue }\lambda_1\approx 8.19. \]

Second principal direction:

\[ \text{eigenvector }v_2\approx \begin{bmatrix}0.51\\-0.86\end{bmatrix}, \qquad \text{eigenvalue }\lambda_2\approx 0.314. \]

Now, we select the first principal direction and project the data there. We keep one coordinate, \(z_i\), and use it to recover the projected point \(p_i\):

\[ \begin{aligned} z_i&=v_1^{\mathsf T}\tilde{x}_i,\\ p_i&=z_iv_1. \end{aligned} \]

Here, \(z_i\) is the one-dimensional representation. The point \(p_i\) lies on our principal direction in 2D.

This projection creates a residual \(r_i\) (the difference between the centered sample and its projection) that is perpendicular, also known as orthogonal, to the principal direction. This orthogonality is important because it guarantees we are projecting to the closest point (we’ll see why).

Finally, we add back the original mean:

\[ \widehat{x}_i=\bar{x}+p_i. \]

Our reconstruction looks like this.

Original two-dimensional data and their reconstruction on the first principal direction, with dark gray segments showing selected residuals.
Projection to one component and reconstruction, from the notebook.

So, when we subtract the reconstruction, the mean cancels:

\[ \begin{aligned} x_i-\widehat{x}_i &=x_i-(\bar{x}+p_i)\\ &=(x_i-\bar{x})-p_i\\ &=\tilde{x}_i-p_i\\ &=r_i. \end{aligned} \]

With this setup, if we now take all the residuals, square their lengths, and compute their average, we’ll get—you guessed it—about \(0.314\)! Exactly the eigenvalue (variance of the discarded second principal component). As you might imagine, there’s an explanation for this match.

Minimal Derivation

In the previous section, I claimed that orthogonality gives us the closest projection. Let’s see why.

A: point x, its orthogonal projection p onto subspace U, and another candidate u in U. B: the right triangle whose sides are the residual r, p minus u, and x minus u.
A. Orthogonal projection onto \(U\). B. The corresponding right triangle.

Our goal is to find a point in \(U\) with the closest distance to \(x\). The residual in the image, where it is orthogonal to \(U\), is given by

\[ r=x-p. \]

So \(x=r+p\). There could be other points in the subspace \(U\), like the point \(u\) in image A. Let’s use this \(u\) as a variable and subtract it from both sides.

\[ x-u=r+(p-u). \]

Notice that \(p\) and \(u\) are in \(U\). Since \(U\) is a subspace (closed under addition and scalar multiplication), \(p-u\) is guaranteed to be in \(U\). Also, since \(r\) is perpendicular to \(U\), it follows that \(r\) and \(p-u\) must be orthogonal. Based on this, we can see our last equation in terms of image B, where \(x-u\) is the hypotenuse and \(r\), \(p-u\) are the legs of a right triangle. Using Pythagoras,

\[ \lVert x-u\rVert^2=\lVert r\rVert^2+\lVert p-u\rVert^2. \]

The last term can only be zero or positive, and it becomes zero when \(u=p\). So the squared distance is minimized at \(p\).

Intuitively, any other point in \(U\) will have a longer residual, as it requires moving sideways in \(U\).

Showing why the orthogonal projection is closest might have seemed like a detour. But now we can use that same orthogonality to answer the question of this blog post. Indeed, we can see the decomposition of every centered point as

\[ \tilde{x}_i=p_i+r_i. \]

And since \(p_i\in U\) and \(r_i\perp U\), we have \(p_i\perp r_i\). So we can use Pythagoras directly:

\[ \lVert\tilde{x}_i\rVert^2=\lVert p_i\rVert^2+\lVert r_i\rVert^2. \]

If we now average over all our points (sum and divide by \(n\) on both sides),

\[ \frac{1}{n}\sum_{i=1}^{n}\lVert\tilde{x}_i\rVert^2 =\frac{1}{n}\sum_{i=1}^{n}\lVert p_i\rVert^2 +\frac{1}{n}\sum_{i=1}^{n}\lVert r_i\rVert^2. \]

We can literally interpret those terms as:

\[ \text{total variance}=\text{retained variance}+\text{discarded variance}. \]

Why? Remember that we said centering our data would pay off in this section.

The total variance of our vector-valued data is defined as

\[ \frac{1}{n}\sum_{i=1}^{n}\lVert x_i-\bar{x}\rVert^2. \]

But since the centered data have mean zero,

\[ \text{total variance}=\frac{1}{n}\sum_{i=1}^{n}\lVert\tilde{x}_i\rVert^2. \]

And \(p_i\), \(r_i\) are also centered.

For the projection, with \(P\) the orthogonal projection matrix:

\[ p_i=P\tilde{x}_i, \]
\[ \frac{1}{n}\sum_{i=1}^{n}p_i =\frac{1}{n}\sum_{i=1}^{n}P\tilde{x}_i, \]
\[ \frac{1}{n}\sum_{i=1}^{n}p_i =P\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{x}_i\right), \]

since \(P\) is the same matrix.

But \(\frac{1}{n}\sum_{i=1}^{n}\tilde{x}_i=0\), so

\[ \frac{1}{n}\sum_{i=1}^{n}p_i=P(0)=0. \]

For the residual,

\[ r_i=\tilde{x}_i-p_i=(I-P)\tilde{x}_i. \]

So

\[ \frac{1}{n}\sum_{i=1}^{n}r_i =(I-P)\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{x}_i\right)=0. \]

Therefore,

\[ \underbrace{\frac{1}{n}\sum_{i=1}^{n}\lVert\tilde{x}_i\rVert^2}_{\text{total variance}} =\underbrace{\frac{1}{n}\sum_{i=1}^{n}\lVert p_i\rVert^2}_{\text{retained variance}} +\underbrace{\frac{1}{n}\sum_{i=1}^{n}\lVert r_i\rVert^2}_{\text{discarded variance}}. \]

And we know that PCA directions are eigenvectors of the covariance matrix. But how do their eigenvalues enter this picture?

Let \(v_j\) be one of these directions, with length one. The coordinate of a centered point along this direction is \(v_j^{\mathsf T}\tilde{x}_i\). Since the data are centered, the variance of these coordinates is

\[ \frac{1}{n}\sum_{i=1}^{n}\left(v_j^{\mathsf T}\tilde{x}_i\right)^2. \]

Remember, \(v_j^{\mathsf T}\tilde{x}_i\) is a scalar, so

\[ \begin{aligned} \left(v_j^{\mathsf T}\tilde{x}_i\right)^2 &=\left(v_j^{\mathsf T}\tilde{x}_i\right) \left(v_j^{\mathsf T}\tilde{x}_i\right)^{\mathsf T}\\ &=\left(v_j^{\mathsf T}\tilde{x}_i\right) \left(\tilde{x}_i^{\mathsf T}v_j\right)\\ &=v_j^{\mathsf T}\left(\tilde{x}_i\tilde{x}_i^{\mathsf T}\right)v_j. \end{aligned} \]

Then, since \(v_j\) is the same direction for every point, we can take it outside the sum:

\[ \begin{aligned} &\frac{1}{n}\sum_{i=1}^{n}\left(v_j^{\mathsf T}\tilde{x}_i\right)^2\\ &\quad=\frac{1}{n}\sum_{i=1}^{n} v_j^{\mathsf T}\left(\tilde{x}_i\tilde{x}_i^{\mathsf T}\right)v_j\\ &\quad=v_j^{\mathsf T} \underbrace{\left(\frac{1}{n}\sum_{i=1}^{n} \tilde{x}_i\tilde{x}_i^{\mathsf T}\right)}_{S}v_j\\ &\quad=v_j^{\mathsf T}Sv_j. \end{aligned} \]

The sum in parentheses is our covariance matrix \(S=\frac{1}{n}X_c^{\mathsf T}X_c\).

Finally, remember that \(Sv_j=\lambda_jv_j\) and \(v_j^{\mathsf T}v_j=1\). Therefore,

\[ \begin{aligned} v_j^{\mathsf T}Sv_j &=v_j^{\mathsf T}\left(\lambda_jv_j\right)\\ &=\lambda_j\left(v_j^{\mathsf T}v_j\right)\\ &=\lambda_j. \end{aligned} \]

So each eigenvalue measures the variance along its principal direction.

In our 2D example, the two orthogonal principal directions cover the whole space, so

\[ \text{total variance}=\lambda_1+\lambda_2. \]

Keeping the first component retains variance \(\lambda_1\). The rest is the squared residual error we have been averaging. So our variance split becomes

\[ \lambda_1+\lambda_2=\lambda_1+\operatorname{MSE}. \]

Therefore,

\[ \operatorname{MSE}=\lambda_2=\frac{1}{n}\sum_{i=1}^{n}\lVert r_i\rVert^2. \]

And in general, with this setup, our \(d\) principal directions form an orthonormal basis. If we keep the first \(k\), the residual contains the components we discarded:

\[ r_i=\sum_{j=k+1}^{d}\left(v_j^{\mathsf T}\tilde{x}_i\right)v_j. \]

Since these directions are orthonormal, the cross terms vanish when we square the length:

\[ \lVert r_i\rVert^2 =\sum_{j=k+1}^{d}\left(v_j^{\mathsf T}\tilde{x}_i\right)^2. \]

Then we average over our points:

\[ \begin{aligned} \operatorname{MSE} &=\frac{1}{n}\sum_{i=1}^{n}\lVert r_i\rVert^2\\ &=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=k+1}^{d} \left(v_j^{\mathsf T}\tilde{x}_i\right)^2\\ &=\sum_{j=k+1}^{d} \underbrace{\left(\frac{1}{n}\sum_{i=1}^{n} \left(v_j^{\mathsf T}\tilde{x}_i\right)^2\right)}_{\lambda_j}. \end{aligned} \]

Therefore,

\[ \operatorname{MSE}=\sum_{j=k+1}^{d}\lambda_j, \]

the sum of the eigenvalues of the discarded components.

Visualizing the Lost Variance

We saw how the discarded variance becomes reconstruction error. Let’s now rotate the line we project onto and see how much we lose.

What I Think I Learned

Compression is not a free lunch. We must be aware of what we are sacrificing when doing it. In this post, we focused our attention on PCA and found that the variance we lost was equal to the reconstruction MSE.

It was pleasing to see how orthogonality provides us with the closest projection, and how squared lengths become variance (after centering and averaging).

And at the end, we found our lost variance sitting right there in the discarded eigenvalues, in the MSE, in the average squared length of our residuals—always in front of us.

Open Questions

  • What do we lose in a nonlinear setting, like a VAE?
  • Besides variance, what other features are important to retain in a compression?

References