Epipolar Geometry

Epipolar Geometry

While stereo vision assumes rectified, calibrated cameras, epipolar geometry describes the relationship between any two camera views of the same scene, without rectification.

Let $P$ be a 3D point that projects to $x$ in image 1 and $x^{\prime }$ in image 2. The two camera centers and $P$ define an epipolar plane. Its intersections with the image planes are the epipolar lines $\ell$ and ${\ell }^{\prime }$. For every point $x$ in one image, its corresponding point $x^{\prime }$ must lie along ${\ell }^{\prime }$. This is the epipolar constraint:
$(x^{\prime }{)}^{T}Fx=0,$
where $F$ is the $3\times 3$ fundamental matrix, which encapsulates the relative orientation and position of two uncalibrated cameras. $F$ has rank 2 and 7 degrees of freedom (3×3 up to scale, minus the $\det F=0$ constraint).

When both cameras are calibrated with intrinsics $K$, $K^{\prime }$, the essential matrix
$E=(K^{\prime }{)}^{T}FK=[T{]}_{\times }R$
captures the metric relative pose: rotation $R\in SO(3)$ and translation direction $T\in {\mathbb{R}}^3$ (up to scale). Here $[T{]}_{\times }$ is the skew-symmetric matrix encoding the cross product with $T$. The essential matrix has 5 degrees of freedom and satisfies the essential constraint: its singular values are $\{\sigma ,\sigma ,0\}$.

The epipoles $e$, $e^{\prime }$ are the projections of each camera center into the other’s image. All epipolar lines pass through these two points.

Given $n\ge 8$ feature correspondences $(x_i,x_i^{\prime })$, $F$ is recovered by the 8-point algorithm: stack the linear constraints $(x_i^{\prime }{)}^{T}F{x}_i=0$ into $A\text{vec}(F)=0$ and take the right singular vector of $A$ corresponding to the smallest singular value. Enforcing $\det F=0$ afterwards by zeroing the smallest singular value of the recovered $F$ gives the final estimate.

Intuition

The central question two-view geometry answers is: given a pixel in one image, where can its matching pixel in the other image possibly be? Without any assumptions, the answer is “anywhere.” But the moment we fix the two camera positions, the 3D point that produced the pixel must lie on a single ray — and that ray, reprojected into the second image, becomes a single line: the epipolar line. Searching for a correspondence thus reduces from a 2D search over all pixels to a 1D search along a line.

The fundamental matrix compactly encodes this constraint for every pair of pixels at once. Multiply the pixel in image 1 by $F$ and you get the epipolar line in image 2. Because $F$ is determined by the geometry of the two cameras, recovering it from a handful of point correspondences effectively tells you how the cameras are positioned relative to one another — up to a projective ambiguity when the cameras are uncalibrated, and up to a sign and scale when they are calibrated.

Properties

Fundamental vs. Essential

(Fundamental matrix — uncalibrated) $F$ relates pixels directly: $(x^{\prime }{)}^{T}Fx=0$. Rank 2, 7 DoF.

$F$ requires no calibration knowledge and is recovered from $\ge 8$ pixel correspondences. It contains the epipolar geometry but not metric information — it cannot distinguish “small scene, close cameras” from “large scene, distant cameras” unless intrinsics are known.

(Essential matrix — calibrated) $E$ relates normalized image coordinates ($\hat{x}=K^{-1}x$): $({\hat{x}}^{\prime }{)}^{T}E\hat{x}=0$. Rank 2, 5 DoF.

$E$ factors as $[T{]}_{\times }R$ and encodes metric relative pose. The translation is recovered only up to scale because $E$ is itself defined up to scale ($sE$ gives the same constraint).

(Essential constraint) The singular values of $E$ are $\{\sigma ,\sigma ,0\}$ — two equal nonzero singular values and one zero.

This follows from the factorization $E=[T{]}_{\times }R$: the rank is 2, and the nonzero singular values equal $\Vert T\Vert$. Any $3\times 3$ matrix with this SVD structure is an essential matrix, and vice versa.

(Relation to intrinsics) $E=(K^{\prime }{)}^{T}FK$; equivalently, $F=(K^{\prime }{)}^{-T}E{K}^{-1}$.

Calibrating each view’s pixels to normalized coordinates converts the pixel-space epipolar constraint into the calibrated-space one. Knowing $K,K^{\prime }$ is what elevates $F$ to $E$.

Epipoles and Epipolar Lines

(Epipoles) The epipole $e$ in image 1 is the projection of camera 2’s center, and satisfies $Fe=0$; analogously $(e^{\prime }{)}^{T}F=0$.

Algebraically, the epipoles are the left and right null-vectors of $F$, reflecting its rank-2 structure. Geometrically, they are the vanishing points of the baseline direction in each image.

(Epipolar lines) Given $x$ in image 1, the corresponding epipolar line in image 2 is ${\ell }^{\prime }=Fx$; given $x^{\prime }$ in image 2, the line in image 1 is $\ell =F^T{x}^{\prime }$.

All epipolar lines in one image pass through that image’s epipole. A point’s corresponding match must lie on its epipolar line.

Recovery Algorithms

(Eight-point algorithm for $F$) Stack $n\ge 8$ correspondences into $A\text{vec}(F)=0$ and take the smallest right singular vector of $A$, then enforce $\text{rank}(F)=2$ by SVD thresholding.

Each correspondence $(x_i,x_i^{\prime })$ contributes one linear row $[u^{\prime }u,u^{\prime }v,u^{\prime },v^{\prime }u,v^{\prime }v,v^{\prime },u,v,1]$ where $x_i=(u,v,1{)}^T$ and $x_i^{\prime }=(u^{\prime },v^{\prime },1{)}^T$. After solving for $\hat{F}$, its SVD $\hat{F}=U\text{diag}({\sigma }_1,{\sigma }_2,{\sigma }_3)V^T$ is modified by setting ${\sigma }_3=0$; the projection onto the rank-2 cone gives the final $F$.

(Hartley normalization) Before the linear solve, translate and scale each image’s coordinates so that the centroid is at the origin and the mean distance is $\sqrt{2}$.

This preconditioning step reduces the linear system’s condition number by orders of magnitude and is essential for numerical stability of the 8-point algorithm. Without it, pixel coordinates in the thousands make the $A$ matrix catastrophically ill-conditioned.

(Five-point algorithm for $E$) Uses 5 correspondences and the essential-constraint polynomial system to recover up to 10 candidate essential matrices.

Faster inside RANSAC than the 8-point method (smaller minimal sample). Disambiguation among the up-to-10 candidates is done by checking the sign of the recovered depth on the available points.

Pose Recovery from $E$

(Four-fold ambiguity) Decomposing $E=U\text{diag}(\sigma ,\sigma ,0)V^T=[T{]}_{\times }R$ yields four candidate $(R,T)$ pairs.

The four come from two rotation candidates ($R=UW{V}^T$ and $R=U{W}^T{V}^T$ where $W$ is a fixed $90°$-rotation matrix) and two sign choices for $T$ (the third column of $U$, $\pm u_3$). Three of the four candidates place at least one triangulated point behind one of the cameras; the single candidate with all points in front of both cameras (positive depth, a chirality check) is the true pose.

(Scale ambiguity) The translation recovered from $E$ is only known up to sign and magnitude.

$sE$ for any $s\ne 0$ gives the same epipolar constraint; equivalently, doubling all 3D distances while halving all focal lengths is observationally equivalent. Metric scale requires external information: a known baseline (stereo rig), a known object size, or inertial data.

Practical Use

(Inside RANSAC) Both $F$ and $E$ are estimated robustly using RANSAC because feature matches contain many outliers.

Minimal sample: 8 correspondences for $F$ (8-point), 5 for $E$ (5-point). Inlier test: symmetric point-to-epipolar-line distance $d(x^{\prime },Fx)+d(x,F^T{x}^{\prime })<t$.

(Degenerate configurations) Pure rotation and planar scenes make $F$ (or $E$) unrecoverable.

Under pure rotation there is no baseline, epipolar geometry collapses, and only a homography relates the views. For a planar scene, any linear combination of the plane’s homography and a rank-1 term satisfies the epipolar constraint — $F$ is not unique. These cases must be detected (e.g., by testing whether a homography explains the matches equally well) and handled with alternative estimators.

Examples

8-point Algorithm, End to End

Given 8 pixel correspondences $(x_i,x_i^{\prime })$:

1. Normalize: apply Hartley’s similarity transform $T,T^{\prime }$ so each point cloud has centroid 0 and mean distance $\sqrt{2}$.
2. Build $A\in {\mathbb{R}}^{8\times 9}$ with rows $[u^{\prime }u,u^{\prime }v,u^{\prime },v^{\prime }u,v^{\prime }v,v^{\prime },u,v,1]$ from the normalized points.
3. Solve $Af=0$: take the right singular vector of $A$ corresponding to the smallest singular value; reshape into $\hat{F}$.
4. Enforce rank 2: SVD $\hat{F}=U\text{diag}({\sigma }_1,{\sigma }_2,{\sigma }_3)V^T$, set ${\sigma }_3=0$, recompose.
5. Denormalize: $F=T^{\prime T}\hat{F}T$.

The recovered $F$ satisfies $(x_i^{\prime }{)}^{T}F{x}_i\approx 0$ for each correspondence, with residuals on the order of the pixel noise.

See Also

• 5.1. The Pinhole Camera — the per-camera projection model
• 5.2. Stereo Geometry — the rectified, calibrated, known-baseline special case
• 4.4.1. RANSAC — the robust estimation wrapper applied to the 8-point / 5-point algorithms
• SfM — the multi-view extension
• 3.4. Homographies — the alternative model that replaces $F$ for planar scenes or pure rotation