Differential Geometric Inference in Surface Stereo

Abstract. Many traditional two-view stereo algorithms explicitly or implicitly use the frontal parallel plane assumption when exploiting contextual information, since e.g. the smoothness prior biases towards constant disparity (depth) over a neighborhood. This introduces systematic errors to the matching process for slanted or curved surfaces, errors that can be non-negligible for detailed geometric modeling of natural objects such as a human face. We use contextual information geometrically to avoid such errors. A differential geometric study of smooth surfaces allows contextual information to be encoded in Cartan's moving frame model over local quadratic approximations. The result enforces geometric consistency for both depth and surface normal; the accuracy of our reconstructions argues for the sufficiency of the approximation. In effect Cartan's model provides the additional constraint necessary to move beyond the frontal-parallel assumption in stereo reconstruction.
(a) (b) (c)
Fig. 6. Middlebury dataset results with our compatibilites.
(a)Left (reference) image (b)Right image (c)Ground truth
(d) Potts, Sum-prod., MMSE est. (e) Trunc. Lin., Sum-prod., MMSE est. (f) Potts, Max-prod., MAP est.
(g) Potts, Max-prod., MAP+subpixel (h) Trunc. Lin., Max-prod., MAP est. (i) Our compat., Max-prod., MAP est.
Fig. 9. (a)(b) Left (reference) and right images. (c) Ground truth disparity map. (d) Potts model derived compatibilities (eq. (12)), Sum-product, MMSE estimate. (e) Truncated linear energy model derived compatibilities (eq. (13)), Sum-product, MMSE estimate. (f) Potts model derived compatibilities (eq. (12)), Max-product, MAP estimate. (g) Potts model derived compatibilities (eq. (12)), Max-product, MAP estimate+subpixel refinement. (h) Truncated linear energy model derived compatibilities (eq. (13)), Max-product, MAP estimate. (f) Our general planar surface model derived compatibilities (eq. (15)), Max-product, MAP estimate. Using other compatibilities one obtains stepwise scalloped patterns because of the frontal parallel plane assumption. On the other hand our result has gradual smooth disparity change, indicating the correct reconstruction. Error statistics in Fig. 10.
(a)Left (reference) image (b)Right image (c) Potts, Max-prod., MAP est.
(d) Potts, Max-prod., MAP+subpixel (e) Trunc. Lin., Max-prod., MAP est. (f) Our compat., Max-prod., MAP est.
Fig. 13.(a) Left (reference) image. (b) Right image. (c) Potts model derived compatibilities (eq.~\eqref{E:PsiPotts}), Max-product, MAP estimate. (d) Potts model derived compatibilities (eq.~\eqref{E:PsiPotts}), Max-product, MAP estimate+subpixel refinement. (e) Truncated linear energy model derived compatibilities (eq.~\eqref{E:PsiFrontal}), Max-product, MAP estimate. (f) Our smooth curved surface model derived compatibilities (eq.~\eqref{E:PsiCurved}), Max-product, MAP estimate. As in the previous examples using other compatibilities one obtains stepwise scalloped patterns because of the frontal parallel plane assumption. On the other hand our result has gradual smooth disparity change, indicating the correct reconstruction.
(a) $\Delta d = 1$ pixel (b) $\Delta d = 0.1$ pixel
Fig. 15. Increase disparity quantization does not solve the ``staircase'' problem for standard compatibilities. Shown are results with Potts model derived compatibilities (eq.~\eqref{E:PsiPotts}), Max-product, MAP estimate: (a) Disparity quantization $\Delta d = 1$ pixel; and (b) Disparity quantization $\Delta d = 0.1$ pixel. Notice that ``staircase'' effect still exists even with finer disparity quantization.
(a) Left image (b) Our result
(a) Reconstructed surface normal (b) Zoom in of nose region
Fig. 16. Reconstruction results. (a) Left (reference) image of another face pair. (b) Result using our compatibilities (eq.\eqref{E:PsiCurved}), Max-product, MAP estimate. (c) Reconstructed surface normal. (d) Zoom in of nose region. For display purpose surface normal and depth are displayed once every five pixels in both x and y directions.