Joint Structure-and-Motion Estimation

Even in the 3D–2D approach, the three-dimensional points $\mathbf{x}_i$ are not known exactly but have been obtained through a previous triangulation procedure. Treating them as error-free data therefore artificially separates structure estimation from motion estimation.

A more comprehensive approach is to treat both the camera pose and the three-dimensional point positions as unknowns and to directly minimize the reprojection errors in the images. Under the assumption of independent, isotropic Gaussian errors in the image coordinates, this minimization corresponds to maximum-likelihood estimation:

$$\sum_i \left( \left\Vert \mathbf{p}_1-\hat{\mathbf{p}}_1 \ri... ...\Vert \mathbf{p}'_2-\hat{\mathbf{p}}'_2 \right\Vert^2 \right).$$ (10.101)

The estimated projections can be expressed as
$$\begin{align} \hat{\mathbf{p}}_1 &= \mathbf{K}_1\mathbf{R}_1 (\hat{\mathbf{x}}_i... ...}'_2 &= \mathbf{K}_2\mathbf{R}_2 (\hat{\mathbf{x}}'_i-\mathbf{t}_2), \end{align}$$
to which the constraint given by equation (10.99) must be added, while retaining the actual three-dimensional positions of the points $\hat{\mathbf{x}}_i$ and $\hat{\mathbf{x}}'_i$ at the two instants in time as unknowns.

In this way, both camera motion and the three-dimensional position of each feature are estimated simultaneously. Maximum-likelihood estimation again requires solving a nonlinear problem, but with a number of unknowns that grows with the number of points considered. For a rectified stereo pair, the cost function can be further simplified by exploiting the particular geometry of the configuration.