Visual Servoing

Visual Servoing

Visual servoing is the closed-loop control of a robot using visual feedback from one or more cameras. It is the bridge where computer vision (perception) and robot kinematics (action) meet: the image is the sensor, the joint velocities are the command, and the task is specified directly in terms of what the camera sees.

Let $s\in {\mathbb{R}}^k$ be a vector of image features measured from the current camera view (e.g., pixel coordinates of tracked points, moments of a blob, or a 6-DoF pose recovered via two-view geometry), and let $s^{\ast }$ be the desired feature vector. The visual servoing objective is to drive the error $e=s-s^{\ast }$ to zero.

The feature velocity is related to the camera’s rigid body velocity $V^c=(v,\omega )\in {\mathbb{R}}^6$ by the image (interaction) Jacobian $L_s$:
$\dot{s}=L_s(s,z)V^c,$
where $z$ is the (possibly estimated) depth of each feature.

Combining with the robot’s body Jacobian $J^b$, which maps joint velocities to camera twist (when the camera is rigidly mounted on the end-effector), gives the composite relation
$\dot{s}=L_s{J}^b\dot{\Theta }.$
A proportional control law for exponential error decay is
$\dot{\Theta }=-\lambda (L_s{J}^b{)}^{+}e,$
where $(\cdot {)}^{+}$ is the Moore-Penrose pseudoinverse and $\lambda >0$ is a gain.

Two canonical variants:

• Image-Based Visual Servoing (IBVS): features $s$ are pixel coordinates; the control loop closes in image space. Robust to calibration error but may exhibit unstable 3D camera paths.
• Position-Based Visual Servoing (PBVS): features $s$ are the 3D pose recovered from images via epipolar geometry or pose-from-known-model (PnP); control is in Cartesian space. Geometrically clean but sensitive to calibration and pose-estimation noise.

Intuition

Forward kinematics tells the robot where its hand is given the joint angles. Differential kinematics tells it how a small change in joint angles moves the hand. But neither answers the more practical question: how do I move the hand to where that object is in the image?

Visual servoing closes the loop. The camera — typically mounted on the end-effector (“eye-in-hand”) — produces a stream of image features that change as the robot moves. Differentiating the pinhole projection with respect to camera motion gives the image Jacobian $L_s$, which plays the same role on the vision side that the robot Jacobian $J^b$ plays on the actuation side. Chaining them,
$\text{joint velocity}\stackrel{J^b}{\to }\text{camera velocity}\stackrel{L_s}{\to }\text{feature velocity},$
yields a direct map from what the robot commands to what the camera sees. Inverting that chain (via pseudoinverse) gives the control law.

The deep point is that visual servoing reuses the same Lie-algebraic machinery — twists, adjoints, Jacobians — developed for open-loop kinematics, now in a closed-loop, sensor-driven setting. The image becomes just another coordinate chart on the task.

The Image Jacobian

For a single 3D point $X=(X,Y,Z{)}^T$ in the camera frame, the pinhole projection gives pixel coordinates $(u,v)=(fX/Z,fY/Z)$. Differentiating, and using the rigid body velocity $\dot{X}=-v-\omega \times X$ (point velocity relative to a moving camera), one obtains the classical 2-row-per-point interaction matrix:

$$\left[\begin{array}{cccccc}-1/Z& 0& x/Z& xy& -(1+x^2)& y\\ 0& -1/Z& y/Z& 1+y^2& -xy& -x\end{array}\right],$$

where $(x,y)$ are normalized image coordinates. Stacking such rows for $k$ tracked points produces $L_s\in {\mathbb{R}}^{2k\times 6}$. At least three non-collinear points ($k\ge 3$) are required for $L_s$ to have rank 6.

The depth $Z$ enters $L_s$ and must be estimated — typically from a two-view measurement, a stereo rig, or a constant “desired depth” approximation.

Links to Robotics

• Forward Kinematics — gives the rigid transform of the camera relative to the base, used to bring feature errors into the robot’s reference frame.
• Robot Jacobians — the body Jacobian is composed with the image Jacobian to close the loop from image error to joint command.
• Singularities — $L_s{J}^b$ can become rank-deficient at certain configurations, causing visual servoing to stall or oscillate even when the robot itself is well away from kinematic singularities.
• Jacobian IK — the pseudoinverse control law in visual servoing is structurally identical to velocity IK, with $L_s{J}^b$ playing the role of $J$.

Links to Computer Vision

• 5.1. The Pinhole Camera — the geometric model that produces the image Jacobian via chain rule on the projection equations.
• 5.3. Epipolar Geometry — supplies the relative pose $(R,T)$ that defines the goal in PBVS.
• 5.2. Stereo Geometry — an alternative way to recover the depth $Z$ that enters $L_s$.
• 4. Feature-Based Alignment — the feature detection / matching pipeline supplies the image measurements $s$.
• SfM — PnP-style pose recovery from 2D-3D correspondences is the workhorse of model-based PBVS.

IBVS vs. PBVS

Aspect	IBVS	PBVS
Feature space	pixels	3D pose
Control frame	image	Cartesian
Sensitivity to calibration	low	high
3D trajectory	may be erratic	straight line
Singularities	image Jacobian rank loss	none (by construction)
Depth required	yes (for $L_s$)	implicit in pose recovery

A full treatment of hybrid and partitioned schemes (2.5D visual servoing) is left to the reader; see Chaumette & Hutchinson’s 2006 tutorial.