Stereo Geometry

Stereo Geometry

Stereo vision recovers depth by observing a scene from two cameras of known relative pose. Two cameras separated by a baseline $B$ view the same 3D point $P$; the point projects to pixels in each image, and the offset between those pixels encodes how far $P$ lies from the cameras.

A stereo rig is rectified when the two image planes are coplanar, the optical axes are parallel, and the cameras differ only by a horizontal translation $B$ along the shared $x$-axis. In a rectified pair, epipolar lines become horizontal and aligned between the two images, so a pixel $(u_L,v_L)$ in the left image corresponds to some pixel $(u_R,v_L)$ in the right image with the same vertical coordinate. The horizontal offset
$d=u_L-u_R$
is the disparity. With both cameras sharing focal length $f$ (in pixels) and baseline $B$, the depth of the scene point is
$Z=\frac{fB}{d}.$
Depth is inversely proportional to disparity: nearby points shift a lot between the two images; distant points shift very little; points at infinity produce zero disparity.

Any real stereo pair can be turned into a rectified pair via rectification — a pair of homographies that warp the two images so epipolar lines align horizontally. Once rectified, finding the pixel correspondence reduces from a 2D search over the image to a 1D search along a scanline, which is what makes dense depth estimation tractable.

Disparity and Baseline Trade-off $B$ improves depth resolution (a 1-pixel disparity error produces a smaller depth error when $B$ is large), but makes correspondence harder (wider view-angle difference means more occlusions and less similar local image content). Stereo rig design is a compromise between these two effects.

A larger baseline

Intuition

Hold a finger in front of your face and close one eye at a time. The finger jumps horizontally against the background — the jump is disparity, and its size tells you how close the finger is. The brain fuses the two views into a single depth estimate; a stereo rig does the same thing with calibrated cameras and explicit geometry.

The geometry that turns disparity into metric depth is a similar-triangles argument in the rectified setup: the ray from each camera to the scene point forms a triangle with the baseline, and the two triangles share the scene point’s depth $Z$ as an altitude. Solving for $Z$ gives $Z=fB/d$. The formula fails when $d=0$ (point at infinity — no disparity, no depth constraint) and becomes unreliable when $d$ is small (large depth uncertainty per pixel of disparity error).

Rectification reduces the correspondence problem to scanline matching. Without rectification, to match a pixel in the left image the matcher would have to search along an arbitrary epipolar line — diagonal, off-image, or even a pencil of lines through an epipole. With rectification, every match lives on a horizontal row in the right image, which is why every dense stereo algorithm begins with a rectification step.

Properties

Disparity and Depth

(Depth-disparity law) In a rectified pair with baseline $B$ and focal length $f$, depth is $Z=fB/d$.

The formula follows from similar triangles. A point at camera-frame position $(X,Y,Z)$ projects to $u_L=fX/Z+o_x$ in the left image and $u_R=f(X-B)/Z+o_x$ in the right image, so $d=u_L-u_R=fB/Z$. Inverting gives the depth formula. The formula requires identical intrinsics (or post-rectification intrinsics matched between the two cameras) and pure horizontal translation between the two views.

(Inverse relationship) Depth is inversely proportional to disparity; near points have large disparity, far points have small disparity.

A point at depth $Z_1=2$ m with $f=500$ px and $B=0.1$ m has disparity $d_1=25$ px. A point at $Z_2=20$ m has $d_2=2.5$ px. The nonlinearity means disparity resolution is coarse at long range and fine at short range — a consumer stereo camera can tell $0.5$ m from $0.6$ m easily but cannot tell $50$ m from $60$ m at all.

(Depth uncertainty) A disparity error $\delta d$ produces a depth error $\delta Z\approx Z^2\delta d/(fB)$.

Differentiating $Z=fB/d$ with respect to $d$ gives $dZ/dd=-fB/d^2=-Z^2/(fB)$. A sub-pixel disparity error of $0.5$ px at $Z=10$ m with $f=500$ px and $B=0.1$ m yields $\delta Z\approx 1$ m — one reason stereo depth is often fused with other modalities at long range.

Rectification

(Rectifying homographies) Rectification applies homographies $H_L,H_R$ to the two images so that epipolar lines become horizontal and vertically aligned.

Given the relative pose $(R,T)$ between the cameras, one constructs a new virtual camera frame whose $x$-axis is parallel to $T$ and whose image planes are coplanar. The homographies $H_L=K{R}_L^{\text{rect}}{R}_L^T{K}^{-1}$ and $H_R=K{R}_R^{\text{rect}}{R}_R^T{K}^{-1}$ warp the original images into this common frame. After rectification the fundamental matrix reduces to $F=[e{]}_{\times }$ with epipoles at infinity along the $x$-axis, so all epipolar lines are horizontal.

(Loss of field of view) Rectification typically discards image regions near the borders.

Because rectification reprojects the images onto a new plane, portions of the original sensor map outside the rectified frame and must be cropped or masked. Wide-baseline or strongly-converging rigs can lose more than half the pixels to rectification. Practical implementations pick the rectification rotation to maximize the overlap region.

Stereo Matching

(Block matching) The workhorse correspondence algorithm scores image patches at every disparity along the scanline and picks the best match.

For each pixel in the left image, a patch around it is compared to patches at candidate disparities $d=0,1,2,\dots ,d_{\max }$ along the same row of the right image. The match score is typically sum-of-absolute-differences (SAD), sum-of-squared-differences (SSD), or normalized cross-correlation (NCC). The disparity minimizing (or maximizing, for NCC) the score becomes the pixel’s depth estimate. Patch size trades detail against robustness: $3\times 3$ patches preserve edges but are noise-sensitive; $15\times 15$ patches are smooth but blur depth discontinuities.

(Disparity range) Restricting the search to $[d_{\min },d_{\max }]$ speeds matching and rules out impossible geometry.

Given known minimum and maximum scene depth, one computes the corresponding disparity range and only searches that interval. For a rig with $f=500$ px, $B=0.1$ m, scene depths in $[1,10]$ m give $d\in [5,50]$ px — a narrow band that is much cheaper to search than the full image width.

(Ambiguity and occlusion) Textureless regions produce ambiguous matches, and points visible only in one view have no valid disparity.

A blank wall gives the matcher no signal to lock onto, so block matching returns noise. Points occluded in one camera (behind another object from that viewpoint) have no correspondence and any disparity reported for them is wrong. A left-right consistency check — compute disparity from left to right, and again from right to left, and discard pixels where the two disagree — filters many of these cases.

(Global methods) Graph-cut, semi-global matching (SGM), and belief propagation add smoothness priors that tie neighboring disparities together.

Pure block matching decides each pixel independently and produces noisy, fragmented depth maps. Global methods minimize an energy $E(d)=E_{\text{data}}(d)+\lambda E_{\text{smooth}}(d)$ where the smoothness term penalizes disparity jumps between neighbors except across image edges. SGM approximates the 2D optimization with several 1D dynamic-programming passes and is the de-facto standard on embedded stereo rigs. Learned methods (PSMNet, RAFT-Stereo) replace the handcrafted energy with a neural network.

Stereo Rig Design

(Calibration) A stereo rig is calibrated by estimating both intrinsics $K_L,K_R$ and the relative pose $(R,T)$ between the two cameras.

Standard Zhang-style calibration with a shared checkerboard captured in both cameras recovers all parameters jointly by minimizing reprojection error. The output feeds into rectification, which is precomputed once for a fixed rig and applied per-frame as a lookup table.

(Synchronization) The two cameras must capture simultaneously, or moving objects break the rectified-row assumption.

Hardware triggering (a shared shutter signal) is the clean solution; software synchronization (aligning timestamps) works for slow-moving scenes but fails for anything faster than a few pixels per frame interval.

Examples

Depth from Disparity

A rectified stereo rig has focal length $f=700$ px and baseline $B=0.12$ m. A corresponding pair has left pixel $u_L=412$ and right pixel $u_R=388$. What depth does this imply?

Solution $d=u_L-u_R=412-388=24$ px.

Disparity:

Depth: $Z=fB/d=(700)(0.12)/24=84/24=3.5$ m.

Disparity Resolution at Range

For the same rig, how well can depth be resolved at $Z=2$ m versus $Z=20$ m, given a disparity uncertainty of $0.5$ px?

Solution $\delta Z\approx Z^2\delta d/(fB)$.

At $Z=2$ m: $\delta Z=4\cdot 0.5/84\approx 0.024$ m $=2.4$ cm.

At $Z=20$ m: $\delta Z=400\cdot 0.5/84\approx 2.4$ m.

A 100-fold range increase produces a 100-fold drop in relative depth resolution — the quadratic cost of the inverse-disparity formula.

Baseline Trade-off

Doubling the baseline from $B=0.1$ m to $B=0.2$ m on a $f=500$ px rig changes which quantities?

Solution $d=fB/Z$ scales linearly with $B$.

Disparity doubles (better resolution):

Depth uncertainty halves at the same disparity error: $\delta Z=Z^2\delta d/(fB)$ scales as $1/B$.

Matching difficulty increases: wider view angle means larger perspective distortion between left and right patches, more occlusions, and more textureless surfaces appearing differently in the two views. The trade-off between geometric resolution and correspondence difficulty is why hand-held stereo (narrow baseline) and aerial stereo (wide baseline) live at different operating points.

See Also

• 5.1. The Pinhole Camera — the single-camera model that stereo generalizes
• 5.3. Epipolar Geometry — the two-view geometry that rectification simplifies
• SfM — uncalibrated multi-view depth recovery
• MVS — dense depth beyond two views, building on rectified stereo primitives