Introduction

Flow-based generative models underlie many of the most capable modern generative systems (from image synthesis to video generation) and have evolved through a series of innovations, each addressing key limitations of the previous generation. A foundational entry point is normalizing flows, popularized by Rezende and Mohamed [?] , which transform a simple base distribution into a complex data distribution through a sequence of invertible, differentiable transformations. This invertibility enables exact likelihood computation via the change of variables formula, a significant advantage, but early normalizing flows required carefully constrained architectures to keep the Jacobian determinant tractable, limiting their expressiveness. Continuous normalizing flows (CNFs) [?] lifted these architectural constraints by taking the continuous-time limit: rather than a discrete chain of transformations, a CNF parameterizes a smooth vector field whose integral curves carry samples from source to target. This continuous formulation is far more flexible, but training required simulating the full ODE trajectory, which is computationally expensive and difficult to scale. Flow matching [?] resolved this bottleneck with a simulation-free training objective: a neural network is trained to regress directly onto conditional velocity fields, making CNF training practical at scale and allowing the use of arbitrary noise distributions.

Yet a new challenge emerges with flow matching: the trajectories learned by trained models tend to be curved, and accurately simulating these curved paths requires many costly neural network evaluations. For large models—often with billions of parameters—this incurs not just high computational cost but also high latency; in some cases it can take minutes to generate a single sample. This article traces the conceptual arc from normalizing flows through CNFs and flow matching, culminating in rectified flows [?] , a simple approach that straightens the trajectories of flow models to enable significantly faster sampling.

A major culprit behind the high cost incurred when sampling from flow models stems from the geometric properties of the learned flows. It can be challenging to reason about high-dimensional data, but fortunately for us, we can gain an intuition about many of the important geometric properties of flows by visualizing them in low-dimensions. In fact, we can use the exact same algorithms used to train large-scale models to train simple 2D flows on toy distributions and reproduce many phenomena of practical interest. In a related project we developed an interactive web app called Diffusion Explorer [?] that allows users to experiment with training and sampling from flow and diffusion models in 2D.

Sampling from a flow model involves simulating the trajectory of an abstract particle as it moves from random noise to real data by repeatedly querying a neural network to determine the particle's velocity at each point in time. When these trajectories are highly curved, accurately simulating them requires taking many small steps of our expensive neural network. Shown in Figure 2 above, we can see that a flow model trained to generate samples from a simple smiley face distribution produces trajectories that are curved. Understanding where this curvature comes from—and how to eliminate it—is the destination of the narrative arc this article traces. We begin with the foundations of normalizing flows and continuous normalizing flows, build up to flow matching as a scalable training framework, and then show how rectified flows [?] straighten these trajectories to enable fast, low-step sampling.

Normalizing Flows

What is a Normalizing Flow?

A normalizing flow [?] is a generative model that transforms a simple probability distribution, one that is typically easy to sample from, such as a multivariate Gaussian, into a complex target distribution through a sequence of invertible, differentiable transformations. The name reflects the idea that this sequence "flows" samples from a simple ("normal") distribution toward the complex data distribution. Concretely, given a base random variable $z_0 \sim p_0$ drawn from a simple distribution $p_0$, we apply a chain of transformations $f_1, f_2, \ldots, f_K$ to obtain $x = f_K \circ \cdots \circ f_1(z_0)$.

1. Invertible. Invertibility ensures that probability mass is conserved—no mass is created or destroyed as we transform our distribution. Every point in the source distribution maps to exactly one point in the target distribution, and vice versa. This bijective property is what allows us to precisely track how probability density changes as we apply the flow.
2. Differentiable. Differentiability enables us to compute how much the transformation locally stretches or compresses space. This local volume change, captured by the Jacobian determinant, is the key ingredient for computing likelihoods under the transformed distribution.

Evaluating the Data Likelihood

A central advantage of normalizing flows over many other generative models is their ability to compute exact likelihoods. Evaluating the density of a point under a simple distribution like a Gaussian is straightforward—we have a closed-form expression. But evaluating the density under a complex data distribution is not so easy. Normalizing flows solve this problem by providing a principled way to relate the density of a data point $x$ under the complex distribution to the density of its preimage $z = f^{-1}(x)$ under the simple base distribution, using the change of variables formula.

The Change of Variables Formula. The change of variables formula provides the mathematical link between the density of the base distribution and the density of the transformed distribution. For an invertible transformation $f$ that maps $z$ to $x = f(z)$, the density of $x$ is given by:

The term $\left| \det \frac{\partial f}{\partial z} \right|$ is the absolute value of the determinant of the Jacobian matrix of $f$. It measures how much the transformation locally stretches or compresses volume. When the transformation expands a region of space, the density must decrease proportionally (and vice versa) so that the total probability mass is conserved. It is often convenient to work with the log form:

Jacobian Measures Local Volume Change. To build further intuition for the change of variables formula, we can visualize what the Jacobian determinant is actually measuring. The determinant of the Jacobian $\left| \det \frac{\partial f}{\partial z} \right|$ captures the local volume change induced by the transformation $f$. Imagine a small region of space around a point $z$—the Jacobian tells us how much this region is stretched, compressed, or rotated as it is mapped to the region around $f(z)$. If the determinant is greater than one, the transformation is locally expanding space (and the density decreases). If it is less than one, space is being compressed (and the density increases).

Composing Multiple Transformations. In practice, a single transformation is rarely expressive enough to bridge the gap between a simple base distribution and a complex data distribution. Instead, normalizing flows compose multiple transformations $f_1, f_2, \ldots, f_K$, each contributing a small change. The change of variables formula extends naturally to compositions—the log-likelihood under the full flow is the log-likelihood under the base distribution plus the sum of log-determinants at each stage:

Computing the Likelihood of Data. To compute the likelihood of an observed data point $x$, we map it backward through the inverse transformations $f_K^{-1}, \ldots, f_1^{-1}$ to recover its representation in the base space. At each step, we accumulate the log-determinant of the Jacobian of the inverse transformation. This gives us the log-likelihood of the data point:

Maximum Likelihood Training. With the ability to compute exact log-likelihoods, we can train normalizing flows by maximizing the log-likelihood of observed data. Given a dataset $\mathcal{D}$, we optimize the parameters $\theta$ of our flow transformations to maximize:

For each data point, we invert the flow, evaluate the base distribution density, and accumulate the log-determinant corrections. This provides an exact gradient signal for training, in contrast to models that rely on approximate inference.

Jacobian Determinants Are Expensive

While normalizing flows provide an elegant framework for exact likelihood computation, there is a major practical obstacle: computing the determinant of the Jacobian matrix is expensive. For a transformation in $d$ dimensions, computing the full Jacobian determinant requires $O(d^3)$ operations in general:

This cubic cost is prohibitive for high-dimensional data like images, where $d$ can be in the tens of thousands or more. A substantial body of work has addressed this by restricting the form of the transformations $f_i$ to make the Jacobian determinant cheaper to compute—for example, planar flows, Real NVP, and Glow use architectures with triangular Jacobians, reducing the determinant to a product of diagonal entries. However, these restrictions limit the expressiveness of the flow. An alternative approach, which we discuss next, is to move to a continuous-time formulation that avoids the Jacobian determinant entirely.

Continuous Normalizing Flows

Rather than composing a fixed number of discrete transformations, continuous normalizing flows (CNFs) (Chen et al., 2018) replace the sequence of layers with a single continuous-time ordinary differential equation (ODE) parameterized by a neural network. Instead of applying $K$ separate transformations, we define a smooth trajectory through the space of probability distributions.

A flow model learns to bridge a simple source probability distribution $p$ that is easy to draw samples from, like a multivariate Gaussian $\mathcal{N}(0, \sigma^2 I)$, to a complex data distribution $q$ by defining a continuous transformation between the two. We define a continuous sequence of probability distributions, called a probability path $(p_t)_{0 \leq t \leq 1}$, that smoothly interpolates between our simple source distribution $p_0$ and our data distribution $p_1 = q$ (see Figure 11). We index this path by a time variable $t \in [0, 1]$, where $t=0$ corresponds to the source distribution and $t=1$ corresponds to the target distribution. By drawing samples from $p_0$ and transforming them over time we can produce samples distributed according to our data distribution $p_1 = q$.

Sampling Trajectories

Individual samples drawn from the source distribution $p_0$ have trajectories $x(t)$ that trace a path from $x_0 \sim p_0$ to $x_1 \sim p_1$ as time progresses from $t=0$ to $t=1$. These trajectories are the paths that individual "particles" take as the continuous normalizing flow transforms the source distribution into the target distribution. Understanding these trajectories is important because the geometry of these paths—how straight or curved they are—will turn out to have significant implications for the efficiency of sampling.

CNFs Learn Velocity Fields

Rather than directly modeling the flow $\psi_t(x)$, continuous normalizing flows model a time-dependent velocity field $v_t(x)$ that "generates" the flow. By taking this velocity field we can solve a set of ordinary differential equations (ODEs) to recover the flow, in a process called simulation. By starting from some initial point $x$ at time $t=0$, we can trace the trajectory of this point over time according to the velocity field $v_t(x)$ using the following ODEs

The solution to these ordinary differential equations involving $v_t(x)$ is itself the flow $\psi_t(x)$. There are a variety of numerical methods for simulating these ODEs which approximate the continuous trajectory by taking a series of discrete steps. Perhaps the simplest such method is Euler's method, which approximates the trajectory of the flow by taking small linear steps in the direction of the velocity field at each time step $x_{t + \Delta t} = x_t + \Delta t \cdot v_t(x_t)$.

Training Efficiency

A key advantage of the continuous-time formulation is that it enables more efficient likelihood computation. Recall that discrete normalizing flows require computing the full Jacobian determinant at each layer, which costs $O(d^3)$. In the continuous setting, the instantaneous change of variables formula replaces the expensive determinant with a much cheaper trace:

The trace of the Jacobian is only $O(d)$, a dramatic improvement over the $O(d^3)$ cost of the full determinant. This makes CNFs practical for higher-dimensional data where discrete normalizing flows with unrestricted architectures would be computationally prohibitive.

Despite this efficiency gain, likelihood-based training of CNFs still has a significant drawback: it requires solving an ODE at every training step. To compute the log-likelihood of a data point, we must integrate the trace of the Jacobian along the entire trajectory from $t=0$ to $t=1$, which requires solving the ODE for each training example. This simulation is computationally expensive and becomes a bottleneck during training.

Flow Matching

Now that we have discussed the foundations of normalizing flows and continuous normalizing flows, we can discuss flow matching—a simulation-free training method that avoids the expensive ODE solving required by likelihood-based training. Please check out [?] for a more thorough introduction.

The motivation behind flow matching is to be able to learn our vector field $v_t(x)$ without having to do expensive simulation, meaning without having to use Euler integration or some other technique to solve ODEs. Flow matching allows us to learn $v_t(x)$ by solving a simple regression loss!

Flow matching can be broken down into two key steps:

1. We need to define our probability path $p_t(x)$ for interpolating between our source $p$ and target distribution $q$.
2. We need to train a velocity field $v_t^\theta(x)$ that generates the path $p_t$ through regression.

Defining the Probability Path

We will focus on a specific choice of probability path called the linear path. The linear path can be defined through a simple linear interpolation between our source and target distributions:

In the examples I provide throughout this article, our source distribution $p_0$ is always a standard Gaussian distribution $p_0(x) = \mathcal{N}(x|0, I)$, and our target distribution $q$ is a complex 2D distribution representing a smiley face. However, in general, flow matching affords much more flexibility in the choice of probability paths and source distributions.

Regressing the Velocity Field

Now, the second step of flow matching is to "match" the true velocity field $v_t(x_t)$ with an approximation $v_t^\theta(x_t)$, parameterized by a neural network, by optimizing a simple regression objective.

However, there is a catch: we do not have direct access to the true velocity field $v_t(x_t)$! $v_t(x_t)$ is difficult to directly construct in practice as it governs the transformations between two jointly distributed high dimensional distributions. So, how can we optimize this objective?

Luckily, we can create a related but much simpler objective by conditioning our velocity field on a particular instance from our target distribution $x_1 \sim q$. This yields the conditional velocity field $v_t(x_t | x_1) = \frac{x_1 - x_t}{1 - t}$.

Equipped with this conditional vector field, we can create a regression objective called conditional flow matching.

If we then plug in our specific conditional velocity field for our choice of a linear probability path, we get the remarkably simple training objective:

Incredibly, the conditional flow matching and the flow matching objectives have the same gradients $\nabla_\theta \mathcal{L}_{CFM}(\theta) = \nabla_\theta \mathcal{L}_{FM}(\theta)$ , meaning we can optimize our tractable conditional flow matching objective and solve the flow matching problem. During training we simply need to draw pairs $(x_0, x_1)$ from our source and target distributions, interpolate between them to get $x_t$, and then train our velocity field $v_t^\theta(x_t)$ to predict the straight-line velocity $x_1 - x_0$.

A critical fact that is worth emphasizing, is that we are matching the conditional velocity $v_t(x_t|x_1)$ which is conditioned on the target point $x_1$ with our learned velocity field $v_t^\theta(x_t)$ which only "knows" about the current $x_t$. If we were to condition our learned vector field on $x_1$ as well, then the problem would become trivial as the model could just predict some scaled version of $x_1 - x_t$. So, the model $v_t^\theta(x_t)$ has to identify the likely destination $x_1$ using only the information about the location $x_t$ at time $t$.

Stochastic Interpolants

Stochastic interpolants [?] generalize the flow matching framework by adding controlled noise to the interpolation path. Rather than following a purely deterministic linear path between source and target, stochastic interpolants introduce a noise term $\sigma_t \cdot \varepsilon$ where $\varepsilon \sim \mathcal{N}(0, I)$:

The noise schedule $\sigma_t$ controls how much stochasticity is introduced at each time step. When $\sigma_t = 0$ for all $t$, we recover the deterministic flow matching setting. When $\sigma_t > 0$, the interpolant becomes stochastic, bridging the gap between deterministic flow models and stochastic diffusion models. This unifying perspective reveals that many seemingly different generative modeling approaches are special cases of a single framework.

Two Frameworks, One Idea. Remarkably, the flow matching framework [?] and the stochastic interpolants framework [?] were developed independently and in parallel, arriving at the same core insight: that one can train continuous normalizing flows by regressing onto conditional velocity fields without simulation. Despite different mathematical formulations and notation, both frameworks provide simulation-free training objectives that are equivalent under appropriate choices of interpolation schedules.

Curvature is the Enemy of Speed

With the fundamentals of flow models and flow matching established, we can now investigate some of their idiosyncrasies—and how they come up in practice. We showed above that the trajectories produced by a flow model trained with flow matching are curved (see Figure 2). To further illustrate this point, if we superimpose the source and target distributions we can see that this curvature is even more extreme (see Figure 19).

An astute reader might recall that we trained our velocity field $v_t^\theta(x)$ to match straight trajectories $X_1 - X_0$ due to our choice of a linear path. So why does our model then learn curved trajectories, and why is this an issue? Answering the latter question—why curvature is a problem—is more straightforward: the answer is speed.

When drawing new samples from a flow model we perform numerical integration using the trained velocity field $v_t^\theta(x_t)$. At their core, numerical integration algorithms like Euler's method involve making finite steps in the direction of the velocity field: $x_{t + \Delta t} = x_t + \Delta t \cdot v_t^\theta(x_t)$. We are making local linear approximations of the "true trajectories". The degree to which this approximation is accurate depends on how curved the trajectories are, and the size of steps we can take without deviating from the true trajectory, degrading sample quality.

The punch line: curvature is the enemy of speed. Highly curved trajectories are challenging to accurately simulate with a small number of steps. This means we need to make many calls to our large neural network representing our vector field $v_t^\theta(x)$ in order to accurately approximate these trajectories, leading to high latency and computational cost. But why does our model learn these curved trajectories in the first place? The answer has to do with how our source and target random variables are jointly distributed, a concept called a coupling.

What is a Coupling?

When training our velocity field $v_t^\theta(x)$ with flow matching, we need to draw pairs $(x_0, x_1)$ from our source and target distributions $p_0$ and $q$. Something that we glossed over a bit in the section about Flow Matching is how exactly we should draw these pairs. This is actually a crucial design choice, called a coupling, that has a significant impact on the geometry of the learned flow, and is the key culprit behind our curved trajectories.

A coupling is the joint distribution $\pi(x_0, x_1)$ between our source and target random variables. This coupling dictates how our pairs $(x_0, x_1)$ used during training are distributed. The key requirement of a coupling is that the marginals are the source $\pi(x_0) = p$ and target distributions $\pi(x_1) = q$.

The simplest form of coupling, and the one we investigate in this article, is an independent coupling (see Figure 21), where we independently draw $X_0 \sim p$ and $X_1 \sim q$, and we have that $\pi(x_0, x_1) = \pi(x_0)\pi(x_1)$. This allows us to trivially construct pairs $(x_0, x_1)$ during training, and is a natural choice in scenarios where we don't have any known structure associating pairs from our source and target distributions.

As mentioned above, our choice of independent coupling is the key culprit behind our curved trajectories. You can see in Figure 21 that the lines connecting independently drawn source and target points cross each other a lot. These intersections lead to curved trajectories because they introduce branches in our paths that our learned velocity field $v_t^\theta(x)$ can not resolve.

An alternative to the independent coupling is an optimal transport coupling (see Figure 22), which connects source and target points in a way that minimizes the overall cost of transporting mass from the source to the target distribution. This coupling tends to produce fewer crossing paths, which leads to straighter trajectories. However, optimal transport couplings are more challenging to compute, especially in high dimensions, and so they are less commonly used in practice.

Paths Crossed at the Wrong Time

Our learned flow model $v_t^\theta(x)$ is not capable of accurately modeling the crossing paths produced by our independent coupling; this incapability manifests itself in curved trajectories. More precisely, say two paths formed by the pairs $(x_0^a, x_1^a)$ and $(x_0^b, x_1^b)$ intersect at some point $x$ at time $t$, or at least nearly intersect. This results in two distinct velocities $x_1^a - x_0^a$ and $x_1^b - x_0^b$ that our learned velocity field $v_t^\theta(x)$ is supposed to match at the same location $x$ and time $t$. This is not possible because our learned velocity field $v_t^\theta(x)$ is only a function of the current location $x$ and time $t$.

Our learned velocity field $v_t^\theta(x)$ cannot accurately predict both desired velocities at this intersection point, and so it ends up predicting the average of these two velocities. This is also true more generally, whenever we have many paths intersecting in a small neighborhood. Our learned velocity field averages out the conflicting velocities by taking the conditional expectation of velocities passing through this point: $\mathbb{E}[X_1 - X_0 | X_t = x]$. Finally, because the average velocity at these intersection points can change as we move through space, we develop curved trajectories. So, despite the fact that we train our flow model to match straight-line velocities, we end up with curved trajectories.

Rectified Flows

We have discussed why curved trajectories are difficult to simulate with a small number of steps, and now we also understand why a flow model learns curved trajectories when using an independent coupling. Now we ask the question: how can we learn straighter trajectories? A solution to this problem is exactly what Rectified Flows provide us with, and given all of the context above it is actually a startlingly simple solution laying in plain sight.

The Algorithm

Rectified flows straighten out the trajectories of flows by replacing the naive independent coupling used in vanilla flow-matching training with one induced by the model itself. First, we train a model with flow matching using an independent coupling. Next, we generate new pairs $(X_0, X_1^1)$ by drawing $X_0 \sim p$ and applying our learned flow model to get $X_1^1 = \psi_1^1(X_0)$. This new coupling $\pi_1 = (X_0, X_1^1)$ is then used to retrain a new flow model $v_\theta^2$. By repeating this process multiple times we can progressively straighten out the trajectories of our flow model. The full procedure is outlined in the algorithm below.

Why it Works

We draw samples from our trained flow model by solving an ordinary differential equation of the form

This forms a deterministic flow, where it is guaranteed that trajectories $\psi_t(x)$ are unique (under some mild regularity conditions). This uniqueness property is crucial to understanding why rectified flows work. The uniqueness of trajectories in deterministic flows means that two distinct trajectories cannot intersect at the same point in space and time $(x,t)$. If this did happen, then the two trajectories would have to coincide for all times, contradicting the assumption that they are distinct. Deterministic flows therefore forbid crossing, branching, or merging of trajectories. The deterministic nature of these flows is inherited by the coupling induced by integrating the flow.

When we generate new pairs $(X_0, X_1^k)$ by flowing samples from our source distribution through our learned flow model, we are guaranteed to get a coupling where trajectories do not intersect. By retraining on this coupling, we are effectively removing the conflicting velocities at intersection points that caused curvature in the first place.

Comparisons

We can also compare the trajectories learned by a standard flow matching model versus a rectified flow model (see Figure 25 ). The rectified flow model learns significantly straighter trajectories, which are easier to simulate with fewer steps.

This difference in curvature has a direct impact on how many steps are needed during sampling. We can observe this effect by comparing how well Euler's method approximates the "ground truth" trajectory (using many steps) with varying numbers of integration steps (see Figure 26 ). Notice how the rectified flow model produces accurate approximations even with very few steps, while the flow matching model's curved trajectories lead to significant deviation from the true path.

Finally, we can compare the vector fields learned by a standard flow matching model versus a rectified flow model (see Figure 27 ). The rectified flow model learns vector field that is more consistent over time, meaning the model has lower curvature in its trajectories.

Acknowledgements

I'd like to acknowledge my friend Sebastián Gutiérrez Hernández for his valuable feedback on this project, particularly on the formal explanations presented in this article. I would also like to thank Benjamin Hoover, Polo Chau, and Vivek Anand for their feedback on the visualizations and writing.

References

How to Cite

@article{helbling2026flowsurvey,
  title   = {A Visual Survey of Flow-Based Generative Models},
  author  = {Helbling, Alec},
  year    = {2026},
  url     = {https://alechelbling.com/qualifier-writeup}
}