Multi-View Stereo (MVS)

Multi-View Stereo

Multi-View Stereo (MVS) is the dense reconstruction step that follows Structure from Motion. Given images $I_1,\dots ,I_n$ and their already-calibrated poses $(K,R_i,T_i)$ from SfM, MVS recovers a dense 3D model — ideally a depth value at every pixel, or a surface mesh covering the scene — rather than a sparse cloud of feature points.

The canonical problem: for each pixel $x$ in a reference view, find its depth $z$ such that the 3D point $X=z{K}^{-1}\tilde{x}$ (in camera coordinates) projects consistently into the neighbor views that see it. Consistency is measured by photometric similarity (matching image patches) combined with geometric regularization (smooth surfaces, visibility).

Two families of algorithms dominate:

• Depth-map fusion. For each reference view, compute a dense per-pixel depth map by searching along epipolar lines in the neighbors, then fuse all depth maps into a single 3D model. PatchMatch MVS (COLMAP, OpenMVS) is the standard modern variant.
• Volumetric methods. Discretize the scene into a voxel grid and carve, color, or level-set the surface so that voxels consistent with all views are kept. Space carving and Poisson reconstruction are representative.

MVS outputs are typically post-processed by meshing (Delaunay, Poisson, or TSDF fusion) and texturing (projecting source images onto the mesh).

Intuition

SfM recovers where the cameras are; MVS recovers everything else. With camera poses known, the per-pixel depth problem decouples into many small triangulations — one per reference pixel — each resembling the two-view stereo problem, but now drawing evidence from many neighbor views instead of just one.

The trade-off is between photoconsistency and regularization. Matching small image patches along the epipolar line is inherently ambiguous in textureless regions and near occlusion boundaries; regularization pushes neighboring pixels to have similar depths unless the image evidence strongly disagrees. Every modern MVS algorithm is some balance of these two forces.

The payoff over sparse SfM is a reconstruction suitable for rendering, physical measurement, and robotics — anywhere the surface of the scene is what matters rather than just its visible feature points.

Properties

Depth Map Estimation

(Per-pixel depth search) For each reference pixel, depth is estimated by searching along the epipolar line in neighbor views for the patch most similar to the reference patch.

The search parameter is depth $z$ (or equivalently disparity for rectified pairs). For candidate $z$, the 3D point $X(z)=z{K}^{-1}\tilde{x}$ projects to pixel $x^{\prime }(z)=\pi (P^{\prime }X(z))$ in each neighbor. The depth that maximizes a photoconsistency score is kept.

(Patch-based photoconsistency) Matching is performed over small image patches (e.g., $5\times 5$ or $9\times 9$), using cost functions that are robust to illumination and exposure change.

Normalized cross-correlation (NCC) and census-transform Hamming distance are standard; zero-mean NCC additionally compensates for brightness offsets. SSD alone is rarely used because of its sensitivity to gain and vignetting.

(PatchMatch propagation) Modern dense MVS uses PatchMatch to avoid sweeping through all depth hypotheses.

Random initial depth and normal per pixel, then iteratively propagate good hypotheses to neighbors (exploiting spatial coherence) and refine them with random perturbations. Converges in a handful of iterations and is the core of COLMAP’s dense stage.

View Selection and Visibility

(Neighbor view selection) Not all cameras are useful for every pixel — occlusion and grazing angles degrade matching.

For each reference view, a few (typically 5–10) source views are chosen with favorable baselines (for triangulation accuracy) and overlapping fields of view (for visibility). Scores balance baseline angle, pixel overlap, and resolution similarity.

(Visibility reasoning) A surface point may be occluded in some views and visible in others; naive photoconsistency pooled over all views fails at occlusion boundaries.

Robust MVS methods weight each source view by its estimated visibility: pixels whose reprojection differs strongly from the reference are downweighted, letting the depth estimate trust only views that actually see the point.

Depth Map Fusion

(Cross-view consistency filtering) Per-view depth maps are filtered by enforcing that a point visible from two views agrees on depth to within a tolerance.

Inconsistent estimates are rejected, which removes most matching errors before fusion. The surviving estimates form a dense point cloud in the world frame.

(Surface extraction) The fused point cloud is converted to a mesh by Poisson reconstruction, ball-pivoting, or TSDF integration.

Poisson reconstruction fits an implicit surface whose gradient matches oriented normals; TSDF integration (the basis of KinectFusion) stores a truncated signed distance in a voxel grid and extracts the zero level set via marching cubes.

Volumetric Alternatives

(Space carving) A voxel grid is initialized filled, then each voxel is tested for color-consistency across visible views and removed if inconsistent.

What remains is the photo hull: the maximal scene consistent with the images. Space carving handles arbitrary topology and visibility but scales poorly with resolution compared to depth-map fusion.

(Volumetric level sets) A volumetric energy is minimized with an implicit surface representation, trading off photoconsistency and surface smoothness.

These methods produce watertight surfaces and are popular in small-scale, high-quality reconstruction (e.g., museum artifacts).

Practical Considerations

(Textureless regions) Photoconsistency is ambiguous where the scene has uniform texture; depth maps must fall back to smoothness priors or leave holes.

This is the MVS analog of the aperture problem — matching a patch against many identical patches along the epipolar line yields no signal. Most pipelines mark such pixels with low confidence and interpolate across them during meshing.

(Specular and non-Lambertian surfaces) Photoconsistency assumes a view-independent surface appearance; highlights and mirrors violate it.

Robust costs and median aggregation help, but fundamentally non-Lambertian surfaces require either polarization or neural representations to reconstruct accurately.

(Production systems) COLMAP, OpenMVS, and Meshroom are common open-source pipelines; commercial systems include RealityCapture and Metashape.

Modern neural alternatives (NeRF, 3D Gaussian Splatting) replace the explicit depth-map stage with a learned scene representation but still depend on SfM for the camera poses.

See Also

• Structure from Motion — the sparse reconstruction that provides camera poses to MVS
• 5.2. Stereo Geometry — the two-view version of the dense matching problem
• 5.3. Epipolar Geometry — constrains the matching search to a line