← All projects

Research project

Calibration & Active 3D Sensing

Use informative projected light to establish correspondence, calibrate devices, and recover scene depth.

The problem

Active 3D sensing depends on identifying where projected light is observed. Correspondence errors, imperfect calibration targets, and ambient light can all affect these measurements. This project develops a pipeline for projector-camera correspondence, robust device calibration, and adaptive color structured light for shape reconstruction.

Projector-camera calibration software and recovered surface geometry.
From projected correspondences to calibrated devices and reconstructed geometry.

Framework development

  1. 1. Measure correspondence in one shot

    Obtain projector-camera correspondences with one structured-light image per board pose.

    Single-shot-per-pose (ISMAR-Adjunct'18)

  2. 2. Calibrate robustly

    Match color-coded patterns and jointly refine device parameters and measured correspondences.

    ProCamCalib (TASE'21)

  3. 3. Adapt illumination

    Choose separable projected colors and detect them using ambient-light and device priors.

    Adaptive color SL (ISMAR'23)

From correspondences to calibrated geometry

Single-shot-per-pose measurement

The initial method uses a color-coded spatial structured-light pattern to establish correspondences between the camera image and the projector image. Requiring only one projected pattern per calibration-board pose makes acquisition practical, including when the planar target is imperfect.

Robust calibration and joint refinement

The developed system uses graph-theory-based correspondence matching for an initial calibration, followed by bundle adjustment to jointly refine the device parameters and correspondence coordinates. This addresses measurement noise and imperfect target planarity. The open-source GUI supports the calibration workflow and subsequent reconstruction.

Single-shot-per-pose acquisition and projector-camera calibration in the software interface.
Single-shot-per-pose calibration in the software interface.

Adapting the measurement to the environment

Adaptive color structured light improves the measurements used by the calibration and reconstruction pipeline. Ambient light and device color responses can make projected colors difficult to distinguish. The method selects colors that remain separable in the current setup, then uses maximum a posteriori (MAP) color detection with ambient-light and device priors to estimate correspondences more reliably.

This adaptation does not require preliminary geometric calibration or device response-function calibration. The resulting color measurements support both projector-camera calibration and shape reconstruction.

Adaptive color structured light connects color selection, correspondence detection, calibration, and shape reconstruction.
Adaptive illumination strengthens the measurements used for calibration and reconstruction.

Calibration software demonstration

Papers and implementations

Teaser for A Single-Shot-Per-Pose Camera-Projector Calibration System for Imperfect Planar Targets

A Single-Shot-Per-Pose Camera-Projector Calibration System for Imperfect Planar Targets
Bingyao Huang, Samed Ozdemir, Ying Tang, Chunyuan Liao, Haibin Ling
IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2018
Paper Project Code

Teaser for A Fast and Flexible Projector-Camera Calibration System

A Fast and Flexible Projector-Camera Calibration System
Bingyao Huang, Ying Tang, Samed Ozdemir, Haibin Ling
IEEE Transactions on Automation Science and Engineering (TASE), 2021
Paper Project Code Video

Teaser for Adaptive Color Structured Light for Calibration and Shape Reconstruction

Adaptive Color Structured Light for Calibration and Shape Reconstruction
Xin Dong, Haibin Ling, Bingyao Huang
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2023
Paper Project Code Supplementary

Related research topics and projects

Differentiable ProCam systems recover scene parameters by modeling the forward projection-and-capture process. They connect the geometric measurements studied here with relighting and reconstruction. Projector compensation uses those correspondences and device models to control the appearance of projected content; its calibration requirements vary across methods.

Research topics: 3D Vision & Computational Imaging