Chirality and Reconstruction from Relative Poses

Factorization of the Essential Matrix produces four possible pairs $(\mathbf{R}, \mathbf{t})$, all compatible with the same epipolar geometry.

However, not all these solutions are physically plausible. A three-dimensional point observed by a camera must lie in front of the image plane, that is, it must have positive depth.

The correct solution is therefore the one that, when reconstructing the observed three-dimensional points, gives most of them positive depth with respect to both cameras. This criterion is called the chirality constraint.

Let $(\mathbf{R}, \mathbf{t})$ be one of the four possible decompositions of the Essential Matrix, and let

$$\tilde{\mathbf{m}}_1 = (\tilde{u}_1,\tilde{v}_1,1)^{\top} \qquad \tilde{\mathbf{m}}_2 = (\tilde{u}_2,\tilde{v}_2,1)^{\top}$$ (10.83)

be the normalized camera coordinates of a pair of corresponding points.

Denoting the respective depths by $z_1$ and $z_2$, the three-dimensional coordinates along the two optical rays can be expressed as

$$\mathbf{m}_1 = z_1 \tilde{\mathbf{m}}_1 \qquad \mathbf{m}_2 = z_2 \tilde{\mathbf{m}}_2.$$ (10.84)

The relationship between the two observations is therefore

$$z_2 \tilde{\mathbf{m}}_2 = z_1 \mathbf{R} \tilde{\mathbf{m}}_1 + \mathbf{t}.$$ (10.85)

This equation is identical to the triangulation problem for a point observed from two different views. Once one of the four possible factorizations of the Essential Matrix has been selected, the depths $z_1$ and $z_2$ can be estimated using any triangulation technique, such as those discussed in Section 10.3.1.

Checking chirality simply consists in verifying that both depths are positive:

$$z_1 > 0 \qquad \wedge \qquad z_2 > 0.$$ (10.86)

In practice, the procedure is as follows:

1. generate the four possible decompositions of the Essential Matrix;
2. triangulate a set of corresponding points for each decomposition;
3. count the number of points having positive depth with respect to both cameras;
4. select the decomposition that maximizes this number.

Since the translation vector extracted from the Essential Matrix is known only up to a multiplicative factor, the three-dimensional points reconstructed in this way are likewise determined only up to the same scale.

It is important to note that the chirality constraint does not depend on the particular procedure used to estimate the Essential Matrix, but is a general geometric property of three-dimensional reconstruction from relative poses. The same principle can therefore be applied whenever the relative rotation and translation between two sensors are known.