The problem
A projector-camera system needs a forward model: given a projector image and a scene, what will the camera observe? A useful model must connect device responses, surface geometry and materials, occlusion, and indirect illumination. Differentiability makes it possible to estimate unknown scene parameters and optimize projector inputs through that same model.
From unified mappings to explicit 3D scenes
-
1. Unify project-and-capture tasks
Learn geometric and photometric mappings for relighting, compensation, and reconstruction.
-
2. Make light transport explicit
Use differentiable path tracing to separate scene parameters and simulate multi-bounce illumination.
-
3. Represent the scene in 3D
Use Gaussian geometry and materials for efficient simulation across viewpoints.
A shared model for relighting, compensation, and reconstruction
DeProCams learns the projector-camera geometric and photometric mappings in one end-to-end model. It decomposes the forward process into shading-attribute estimation, rough direct-light estimation, and neural rendering. An epipolar-constrained differentiable projector light mask handles occlusion, while geometric and photometric constraints encourage plausible intermediate estimates.
Once fitted, the model supports three tasks that were often handled separately: relighting, projector compensation, and depth/normal reconstruction. This establishes a shared system model rather than a compensation-only inverse mapping.
Explicit physical light transport
DPCS explicitly models scene parameters and light paths. Its differentiable path-tracing model includes multi-bounce light transport, allowing interreflection and soft shadows to be simulated through physically based rendering. Geometry, materials, and device-related parameters can be estimated from projection observations and then reused to simulate new scenes.
The model separates scene parameters that can be inspected and adjusted. The same forward model supports relighting and compensation.
A scene representation that supports different viewpoints
GS-ProCams uses 2D Gaussians to represent scene geometry and materials. It jointly estimates these properties, projector responses, and a global-illumination component from captured multi-view projections using differentiable physically based rendering.
An explicit 3D representation allows ProCam simulation beyond the single view used by image-to-image mappings. Its goal is efficient view-agnostic projection mapping without an additional co-located light source. The project website linked below contains the full method, comparisons, implementation, and data.
System and applications
The common workflow is project known inputs → capture observations → estimate the forward scene model → use that model for rendering or optimization. The representations differ: DeProCams combines explicit geometric/photometric components with learned rendering, DPCS models physical light paths, and GS-ProCams adds an efficient explicit 3D scene representation.
The overview figure above shows reconstruction and simulation alongside downstream projection tasks. These applications connect this project to both ProCam/SAR and physics-informed 3D vision.
Papers and implementations
These papers document the methods and developments described above. Their authors, publication details, and available resources are listed together here.
DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
Bingyao Huang, Haibin Ling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2021 (🏆Best Journal Paper Award)
Paper Code Supplementary Video
DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
Jijiang Li, Qingyue Deng, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2025
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2025
Paper Project Code
GS-ProCams: Gaussian Splatting-Based Projector-Camera Systems
Qingyue Deng, Jijiang Li, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2025
Also in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2025
Paper Project Code
Datasets
Related work
Projector Compensation solves the inverse appearance-control problem and can use a forward scene model during optimization. Physics-Informed Computational Imaging uses explicit optical models for inference through difficult media. Projector-Based Adversarial Attacks uses differentiable projection simulation to evaluate and optimize patterns before physical projection.
Related research topics and projects
Research topics: ProCam & SAR · 3D Vision & Computational Imaging