Contextual Inference in Contour-Based Stereo Correspondence

Abstract. Standard approaches to stereo correspondence have difficulty when scene structure does not lie in or near the frontal parallel plane, in part because an orientation disparity as well as a positional disparity is introduced. We propose a correspondence algorithm based on differential geometry, that takes explicit advantage of both disparities. The algorithm relates the 2D differential structure (position, tangent, and curvature) of curves in the left and right images to the Frenet approximation of the (3D) space curve. A compatibility function is defined via transport of the Frenet frames, and they are matched by relaxing this compatibility function on overlapping neighborhoods along the curve. The remaining false matches are concurrently eliminated by a model of ``near'' and ``far'' neurons derived from neurobiology. Examples on scenes with complex 3D structures are provided.

Our correspondence algorithm is based on differential geometry and takes explicit advantage of both position and orientation disparities. The basic idea is as follows (Fig.2). A curve in $\mathbb{R}^3$ has a tangent, normal, and binormal frame (Frenet frame) associated with every regular point along it. For simplicity, consider only the tangent in this frame, and imagine it as an (infinitly) short line segment. This space tangent projects into a planar tangent in the left image, and a planar tangent in the right image. Thus, space tangents project to pairs of image tangents. Now, consider the next point along the space curve; it too has a tangent, which projects to another pair of image tangents, one in the left image and one in the right image. The key concept that we utilize in this paper is {\em transport}, or the movement of the frame in $\mathbb{R}^3$ from the second point back to the first, which is essentially contextual information expressed geometrically; note that this transport has a correspondence in the left-right image pairs. Our goal is to use this transport to find corresponding pairs of image tangents such that their image properties match, as closely as the geometry can be approximated, the actual space tangents. Two notions of disparity arise from the above transport model. First, the standard notion of positional disparity corresponds, through the camera model, to depth. Second, an orientation disparity is introduced if the space tangent is not in the epipolar plane. In the computational vision literature, orientation disparity is largely unexplored. The success of our system derives, in part, from the simultaneous use of position and orientation disparities, and the underlying differential geometry that naturally combines them.

(a) (b)
(a) (b)
(a) (b) (c)
(a) (b) (c)
(d)
(a) (b) (c)
(a) (b) (c)
(a) (b) (c)
Data
  • [twigL.pgm][twigR.pgm]
  • [lampL.pgm][lampR.pgm]
  • [plantL.pgm][plantR.pgm]
  • [wireL.pgm][wireR.pgm]
  • [perrierL.pgm][perrierR.pgm]
  • [branchL.pgm][branchR.pgm]
  • Related Publications

  • Gang Li and Steven W. Zucker, Contextual Inference in Contour-Based Stereo Correspondence, International Journal of Computer Vision (IJCV), 69(1):59-75, 2006. [SpringerLink], [pdf]

  • Gang Li and Steven W. Zucker, A Differential Geometrical Model for Contour-Based Stereo Correspondence, Workshop on Variational, Geometric, and Level Set Methods in Computer Vision, at IEEE International Conference on Computer Vision (ICCV'03), Nice, France, October 2003. [ps.gz], [pdf]

  • Steven W. Zucker and Gang Li, Geometry of Contour-Based Correspondence for Stereo, Proc. International Symposium on 3D Data Processing Visualization and Transmission (3DPVT'02), Italy, June 2002. [IEEE Xplore], [pdf]