The problem
A projector-camera system needs a forward model: given a projector image and a scene, what will the camera observe? Our models range from learned 2D geometric and photometric mappings to explicit 3D representations of geometry, materials, and light transport. Differentiability allows the model to guide projector-input optimization and, when it represents scene parameters, their estimation from captured images.
From image mappings to explicit 3D scenes
-
1. Learn a 2D forward mapping
Predict camera appearance from projector inputs using learned geometry and photometry in a fixed view.
-
2. Unify project-and-capture tasks
Learn geometric and photometric mappings for relighting, compensation, and reconstruction.
-
3. Make light transport explicit
Use differentiable path tracing to separate scene parameters and simulate multi-bounce illumination.
-
4. Represent the scene in 3D
Use Gaussian geometry and materials for efficient simulation across viewpoints.
A 2D forward project-and-capture model
PCNet, introduced in SPAA, predicts the camera-captured appearance of a projector input using learned 2D geometric warping and photometric interactions with a surface image. It models the forward project-and-capture process in 2D image coordinates for a fixed camera view, without reconstructing explicit 3D geometry or materials.
SPAA differentiates through PCNet to optimize adversarial projector inputs before physical projection. This is a forward-modeling application alongside relighting and compensation; the attack objective is described in Projector-Based Adversarial Attacks.
A shared model for relighting, compensation, and reconstruction
DeProCams learns the projector-camera geometric and photometric mappings in one end-to-end model. It decomposes the forward process into shading-attribute estimation, rough direct-light estimation, and neural rendering. An epipolar-constrained differentiable projector light mask handles occlusion, while geometric and photometric constraints encourage plausible intermediate estimates.
Once fitted, the same model supports relighting, projector compensation, and depth/normal reconstruction.
Explicit physical light transport
DPCS explicitly models scene parameters and light paths. Its differentiable path-tracing model includes multi-bounce light transport, allowing interreflection and soft shadows to be simulated through physically based rendering. Geometry, materials, and device-related parameters can be estimated from projection observations and then reused to simulate new scenes.
The estimated scene parameters can be inspected and adjusted for relighting and compensation.
A scene representation that supports different viewpoints
GS-ProCams uses 2D Gaussians to represent scene geometry and materials. It jointly estimates these properties, projector responses, and a global-illumination component from captured multi-view projections using differentiable physically based rendering.
An explicit 3D representation allows ProCam simulation beyond the single view used by image-to-image mappings. Its goal is efficient view-agnostic projection mapping without an additional co-located light source.
Papers and implementations
SPAA: Stealthy Projector-Based Adversarial Attacks on Deep Image Classifiers
Bingyao Huang, Haibin Ling
IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2022
Paper Code Supplementary Video
DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
Bingyao Huang, Haibin Ling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2021 (🏆Best Journal Paper Award)
Paper Code Supplementary Video
DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
Jijiang Li, Qingyue Deng, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2025
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2025
Paper Project Code
GS-ProCams: Gaussian Splatting-Based Projector-Camera Systems
Qingyue Deng, Jijiang Li, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2025
Also in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2025
Paper Project Code
Related research topics and projects
Projector Compensation solves the inverse appearance-control problem and can use a forward model during optimization. Physics-Informed Computational Imaging extends physical modeling to inference through water and paper envelopes. Projector-Based Adversarial Attacks applies forward models to optimize patterns that change classifier predictions, using PCNet in SPAA and multi-view simulation in VIPA.
Research topics: ProCam & SAR · 3D Vision & Computational Imaging