Changing Variables Without Changing Probability

The Question

Given a continuous random variable \(X\) with density \(f_X\), how could we derive the probability density of its transformation \(g(X)\)? Formally:

Y=g(X), \qquad f_Y(y)=\;?

Short answer: if \(g\) is differentiable and one-to-one, we map \(y\) back to \(x\), evaluate the old density there, and use the inverse derivative to compensate for the transformation's stretching.

If \(x=g^{-1}(y)\), then:

f_Y(y)=f_X(x)\left|\frac{dx}{dy}\right|.

Why I Care

The goal of generative models is to learn how to sample from complex distributions, say images of cats. One approach is to learn how to transform samples drawn from a familiar distribution, e.g., a Gaussian distribution, into our target distribution.

As a concrete example:

X\sim\mathcal{N}(\mu,\sigma^2), \qquad Y=g(X).

We want to find a function \(g\) that transforms a sample from a simple base distribution into data, such as an image of a cat.

I want to understand what happens to density when a transformation stretches or compresses space, as a step toward making sense of normalizing flows.

Setup and Definitions

Let's motivate the math with a concrete case. Imagine a species in which the size of its members' ears determines how good they are at transcribing music.

Usually, the population has similar ear sizes. Even so, a small increase in ear size represents a huge gain in their capacity to transcribe music. Indeed, we can model this by letting

Z\sim\mathcal{N}(0,1)

be the standardized ear size. Here, \(Z=0\) means an average ear size, and a negative \(Z\) means smaller than average—not a physically negative ear. We then let

\[ Y=e^Z \]

be a measurement of how good they are at transcribing music. Our goal is to find the PDF of \(Y\).

Notice that the same event should have the same probability. Density is not probability by itself. In one dimension, it tells us how probability is packed into length. If a transformation stretches that length, the density must shrink so that the total probability stays the same.

For our ear-size example, this means:

P(0.1\le Z\le 0.15) = P(e^{0.1}\le Y\le e^{0.15}).

For this kind of transformation, the general idea is as follows:

Let \(X\) be a continuous random variable with PDF \(f_X\), and let \(Y=g(X)\), where \(g\) is differentiable and one-to-one. If \(x=g^{-1}(y)\), then the PDF of \(Y\) is:

f_Y(y) = f_X(x)\left|\frac{dx}{dy}\right| = f_X\!\left(g^{-1}(y)\right) \left|\frac{d}{dy}g^{-1}(y)\right|.

In \(\mathbb{R}^n\),

\mathbf{Y}=g(\mathbf{X}), \qquad g:\mathbb{R}^n\to\mathbb{R}^n,

and, if \(\mathbf{x}=g^{-1}(\mathbf{y})\),

f_{\mathbf{Y}}(\mathbf{y}) = f_{\mathbf{X}}(\mathbf{x}) \left| \det\left(\frac{\partial\mathbf{x}}{\partial\mathbf{y}}\right) \right|,

where

\frac{\partial\mathbf{x}}{\partial\mathbf{y}} = \begin{bmatrix} \dfrac{\partial x_1}{\partial y_1} & \cdots & \dfrac{\partial x_1}{\partial y_n} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial x_n}{\partial y_1} & \cdots & \dfrac{\partial x_n}{\partial y_n} \end{bmatrix}

is the Jacobian matrix of the inverse transformation.

Minimal Derivation

Let's prove the continuous case when \(g\) is strictly increasing, using the cumulative distribution function (CDF):

\begin{aligned} F_Y(y) &=P(Y\le y) \\ &=P(g(X)\le y) \\ &=P(X\le g^{-1}(y)) \\ &=F_X(g^{-1}(y)). \end{aligned}

Let \(x=g^{-1}(y)\). Differentiating with respect to \(y\) gives:

f_Y(y)=f_X(x)\frac{dx}{dy}.

For decreasing transformations, the inequality reverses in the CDF step, and the final PDF formula uses the absolute value.

We can now solve our motivating problem:

Let \(Z\sim\mathcal{N}(0,1)\) and \(Y=e^Z\). Then \(Y\) is lognormal.

The inverse and its derivative are:

z=\log y, \qquad \frac{dz}{dy}=\frac{1}{y}.

For \(y>0\), we get:

\begin{aligned} f_Y(y) &=f_Z(\log y)\left|\frac{dz}{dy}\right| \\ &=\frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(\log y)^2}{2}\right)\frac{1}{y}. \end{aligned}

So:

f_Y(y) = \frac{1}{y\sqrt{2\pi}} \exp\left(-\frac{(\log y)^2}{2}\right)

for \(y>0\), and \(f_Y(y)=0\) for \(y\le 0\).

The Jacobian here is also interesting. The forward derivative tells us how much a small interval stretches. The inverse derivative in the density formula compensates for that stretch, so the same event keeps the same probability. That is how the Jacobian rescues us. Let's visualize this.

Standard normal density at z approximately 0.69 beside the transformed lognormal density at y equals 2, with corresponding probability intervals shaded — At \(y=2\), the inverse Jacobian is \(1/2\). Locally, the transformation doubles the interval's width, so the density becomes half as high. The corresponding shaded regions still have the same probability.

The polar picture shows the same idea in two dimensions. A \(dr\,d\theta\) rectangle represents a wedge with area \(r\,dr\,d\theta\), so the factor \(r\) is the Jacobian compensating for that change in area. That is why a point uniformly distributed over the unit disk has polar density \(f_{R,\Theta}(r,\theta)=r/\pi\).

Blue wedge in a Cartesian unit disc beside its corresponding red rectangle at radius 0.65 and angle 0.85 radians in polar coordinates — Using a transformation from Cartesian to polar coordinates, the blue wedge and red rectangle represent the same event, so they must carry the same probability.

What I Think I Learned

Under some mild conditions, we can compute the PDF of a transformed random variable. What clicked for me is that the Jacobian is not just a term in the formula: it tells us how much a transformation stretches lengths or areas while preserving probability.

Open Questions

How is this transformation related to linear transformations?
How do normalizing flows use these ideas?
If we know the source and target distributions but not \(g\), can we learn the transformation?

References

Statistics 110, Lecture 22: Transformations and Convolutions—source for the lognormal transformation. The ear-size example is my own.
Blitzstein & Hwang, Introduction to Probability, chapter 8—the book treatment of transformations that accompanies the lecture.
Papamakarios et al., Normalizing Flows for Probabilistic Modeling and Inference (2021)—a review answering the "how do normalizing flows use these ideas?" open question.