The Epipolar Plane

In previous chapters, it was repeatedly noted that a single image cannot provide the world coordinates of the points composing the image without additional information.

Figure 10.1: Epipolar geometry between two cameras: $\mathbf {t}_1$ and $\mathbf {t_2}$ are the pin-holes, $\mathbf {e}_1$ and $\mathbf {e}_2$ are the epipoles, and the world point $\mathbf {x}$ is projected onto the two image points $\mathbf {p}_1$ and $\mathbf {p}_2$, respectively. All the points involved belong to the same plane.

Given the pin-hole camera equation (9.26), the only information that a generic image point $\mathbf {p}$ can provide is a relationship among the (infinitely many) world coordinates $\mathbf {x}$ subtended by the image point, that is, the locus of world coordinates whose projection would produce exactly that particular image point. This relationship is the equation of a line passing through the pin-hole $\mathbf{t}$ and through the point on the sensor corresponding to image point $\mathbf {p}$. Writing equation (9.26) again, it is easy to see the relationship between the parameters of camera i, the image point $\mathbf {p}_i$, and the line representing all possible world points $\mathbf {x}$ subtended by $\mathbf {p}_i$:

$$\mathbf{x} = \lambda (\mathbf{K}_{i}\mathbf{R}_{i})^{-1} \ma... ...t}_{i} = \lambda \mathbf{v}_{i}(\mathbf{p}_i) + \mathbf{t}_{i}$$ (10.9)

where $\mathbf{v}_i$ has the same meaning as in equation (9.27), namely, the direction vector from the pin-hole to the sensor point. As can be inferred both from experience and from the linear relationship linking these points, the subtended point $\mathbf {x}$ is known up to a scale factor $\lambda$.

In the case of stereo vision, there are two sensors, and therefore two reference frames must be defined, with respective parameters $\mathbf{K}_1\mathbf{R}_1$ and $\mathbf{K}_2\mathbf{R}_2$ and pin-hole positions $\mathbf{t_1}$ and $\mathbf {t_2}$, always expressed in world coordinates.

The line (10.9), namely, the locus of world points associated with image point $\mathbf {p}_1$ as seen in the first reference frame, can be projected into the view of the second camera:

$$\begin{array}{rl} \mathbf{p}_2 & = \lambda \mathbf{K}_2 \mat... ...\mathbf{K}^{-1}_1 \mathbf{p}_1 + \mathbf{e}_2 \\ \end{array}$$ (10.10)

where one term varies with the point under consideration and the value $\lambda$, while vector $\mathbf {e}_2$ is constant and independent of the point under consideration.

This constant point is the epipole. The epipole is the intersection point of all epipolar lines and represents the projection of one camera's pin-hole into the image of the other, that is, the “vanishing point” of the epipolar lines.

Given two cameras, the projections of the pin-hole coordinates $\mathbf {t}_1$ and $\mathbf{t}_2$ onto the opposite image are

$$\begin{array}{l} \mathbf{e}_1 = \mathbf{P}_1 \mathbf{t}_2 = ... ...K}_2 \mathbf{R}_2 (\mathbf{t}_1 - \mathbf{t}_2) \\ \end{array}$$ (10.11)

where $\mathbf{P}_1$ and $\mathbf{P}_2$ are the projection matrices. The points $\mathbf {e}_1$ and $\mathbf {e}_2$ are the epipoles. If the definitions of the relative poses expressed in (10.4) are substituted into equation (10.11), the image coordinates of the epipoles, understood as the projection of one camera's pin-hole onto the other image, are

$$\begin{array}{l} \mathbf{e}_1 = \mathbf{K}_1 \mathbf{R}^{\t... ...thbf{t} \\ \mathbf{e}_2 = \mathbf{K}_2 \mathbf{t} \end{array}$$ (10.12)

functions solely of the relative pose between the two cameras.

By construction, matrix $\mathbf{R}$ converts coordinates from camera 1 to camera 2, and $\mathbf{t}$ represents the position of the pin-hole of camera 1 expressed in the reference frame of camera 2.

The lines generated by the points in the first image all pass through the same point formed by the projection of pin-hole $\mathbf {t}_1$ onto the second image: in fact, the point in world coordinates and the two epipoles define a plane (the epipolar plane) containing the possible solutions—the points in camera coordinates—to the three-dimensional reconstruction problem (Figure 10.1).

Epipolar geometry is the geometry relating two images acquired from two different viewpoints. The relationships between the images, however, do not depend on the observed scene, but only on the intrinsic parameters of the cameras and their relative poses.

For each observed point, the epipolar plane is the plane defined by the point in world coordinates and the two optical centers. The epipolar line is the intersection between the epipolar plane and the image plane in the second image. In fact, the epipolar plane intersects the image plane in each image along the epipolar lines and defines the correspondences between the lines.

The following sections discuss both how to derive the line along which a point belonging to one image must lie in another image and how, given two (or more) corresponding points, to obtain the corresponding three-dimensional point.