Multi-View Azimuth Stereo via Tangent Space Consistency

CVPR 2023

Xu Cao, Hiroaki Santo, Fumio Okura, Yasuyuki Matsushita
Osaka University
arXiv Code

3D reconstruction from surface azimuth maps.

Abstract

We present a method for 3D reconstruction only using calibrated multi-view surface azimuth maps. Our method, multi-view azimuth stereo, is effective for textureless or specular surfaces, which are difficult for conventional multi-view stereo methods. We introduce the concept of tangent space consistency: Multi-view azimuth observations of a surface point should be lifted to the same tangent space. Leveraging this consistency, we recover the shape by optimizing a neural implicit surface representation. Our method harnesses the robust azimuth estimation capabilities of photometric stereo methods or polarization imaging while bypassing potentially complex zenith angle estimation. Experiments using azimuth maps from various sources validate the accurate shape recovery with our method, even without zenith angles.

Approach Overview

Teaser

Tangent Space Consistency

An azimuth angle can be converted to a tangent vector with camera orientation. The tangents in different views, but projected from the same surface point, should lie in the same tangent space (i.e., perpendicular to the same normal vector).
Teaser

Neural SDF Optimization

We project the surface point, found by sphere tracing, onto all views, and enforce the surface normal (i.e., SDF gradient) to be perpendicular to all tangents from visible views. We train the neural SDF with this TSC loss, silhouette, and Eikonal loss.

Comparison on DiLiGenT-MV

UA-MVPS

PS-NeRF

MVAS

GT

Reconstrucion with SymPS

Color images (Reference)

Azimuth maps (Input)

Shape

Results on PANDORA

Color images (Reference)

Azimuth maps (Input)

Shape

Related Links

Our work benefits from several works.

  • Our implementation is build upon IDR.
  • DiLiGenT-MV: The multi-view photometric stereo dataset.
  • SDPS-Net: An uncalibrated photometric stereo method for non-Lambertian scenes. We apply this method to obtain the azimuth maps shown in the teaser video.
  • PS-NeRF: A photometric stereo inverse rendering approach for geometry and reflectance recovery.
  • PANDORA: A polarimetric inverse rendering approach for geometry and reflectance recovery. We also used their data for comparison.
  • Symmetric-light PS: An uncalibrated photometric stereo method good at azimuth acquisition.

    • BibTeX

    @inproceedings{mvas2023cao,
      title = {Multi-View Azimuth Stereo via Tangent Space Consistency},
      author = {Cao, Xu and Santo, Hiroaki and Okura, Fumio and Matsushita, Yasuyuki},
      year = {2023},
      booktitle = CVPR,
}