The problem
A projector input rarely appears unchanged on a real surface. Surface texture, geometry, ambient light, and device responses alter the image seen by a viewer or camera. Compensation asks for the projector input that will produce the desired captured appearance. This project follows that inverse problem from learned photometric correction to full compensation, practical deployment, and generalization.
Try it yourself
Photometric compensation corrects colors; geometric compensation aligns pixels.
Follow the projection process
Six real imagesSelect an image to highlight its path and symbols. Blue: original projection. Red: target mapping $\require{HTML}\class{pc-math-target}{x'}=\phi(\class{pc-math-original}{x};A)$ and compensation.
Forward model and demo assumptions
The forward model is $\tilde{x}=\mathcal{T}(\mathcal{F}(x,\tilde{s}))$: photometry $\mathcal{F}$ changes intensities; geometry $\mathcal{T}$ maps them into the camera frame. Map the target $x'=\phi(x;A)$, then compensate it so $\tilde{x}^*\approx x'$.
The browser uses linear RGB, $\mathcal{F}(x,\tilde{s})=r_{\tilde{s}}\odot x+a$, and a four-corner warp for $\mathcal{T}$. Here $r_{\tilde{s}}$ is a surface-response proxy, $\odot$ multiplies channels, and $a$ denotes ambient light. Correction inverts this response, clips available light, and pre-warps pixels.
The projected-gray capture supplies an illustrative response proxy, not calibrated reflectance. Fabric curvature, shadows, scattering, and device responses are omitted. These views are browser simulations; the research networks are not run here.
Inspect the selected image
Surface under projected gray
$\tilde{s}$ is the surface under projected gray illumination—the reference for photometric compensation.
How is the inverse trained?
Project known inputs $x_i$ and record captures $\tilde{x}_i$. The inverse predicts $\hat{x}_i=\pi_\theta^\dagger(\tilde{x}_i,\tilde{s})$. Minimize a reconstruction loss comparing $\hat{x}_i$ with $x_i$—the dashed purple return path.
After training, compensate the target with $x^*=\pi_\theta^\dagger(x',\tilde{s})$. Input–capture pairs provide supervision; ground-truth compensated images are not required.
Explore the two corrections
Illustrative modelTry each correction, then combine them to approach the target.
The simulation requires JavaScript. The six real images above remain available.
Explore the limits of projected light
Projectors have limited brightness and cannot remove ambient light. Enable both corrections, then raise brightness or ambient light to explore these limits.
Framework development
-
1. Learn photometry
Learn how surface appearance and the environment alter projected content.
-
2. Jointly learn geometry and photometry
Connect geometric correction and photometric compensation in one differentiable model.
-
3. Refine the framework and training
Improve photometric interactions and reduce setup-specific training with synthetic pre-training.
-
4. Make deployment practical
Address high-resolution computation, online video adaptation, and real-time compensation.
-
5. Generalize across setups
Separate online geometric correction from photometric compensation to handle unseen setups.
-
6. Use a generative photometric model
Treat photometric compensation as physically constrained, content-aware denoising.
Key ideas
Learning photometric compensation
The starting point is CompenNet: an end-to-end formulation of projector photometric compensation. A U-Net-like backbone and a surface-image subnet learn interactions between the projected content and the camera-captured surface. Multi-level connections carry appearance information through the network. At this stage, the learning problem focuses on photometry, with images geometrically registered.
Bringing geometry into the same model
CompenNet++ extends this formulation to full compensation. A cascaded coarse-to-fine geometric correction subnet learns a sampling grid from projected and captured images, and is connected to the photometric network. Joint training connects alignment and appearance correction in one differentiable pipeline. After training, network simplification improves inference efficiency.
Improving photometry and practical training
The journal extension, CompenNeSt++, retains the joint formulation and refines its two components: WarpingNet handles geometric correction, while CompenNeSt uses a siamese architecture to model the interaction between the surface and projected content. Synthetic-data pre-training reduces the required setup-specific training images and training time. Together, these developments improve the same compensation framework’s model and practical training process.
During training, projected sampling images and their camera captures provide supervision for the inverse mapping. WarpingNet aligns the observations, and the photometric subnet estimates the projector input. The shared project-and-capture formulation avoids requiring ground-truth compensation images for every desired appearance.
After training, the simplified model transforms a desired image into a compensated projector input.
High-resolution, online, and real-time deployment
As the compensation model becomes usable, the next constraint is the deployment loop. CompenHR improves high-resolution compensation through attention-based geometric refinement and a sampling scheme that reduces computation. ViComp changes the acquisition and update process: projected video frames provide training observations while compensation, capture, and short- and long-term model updates run in parallel. CompenRT addresses real-time high-resolution compensation using low-resolution inputs and an efficient photometric network with upsampling and attention.
Generalizing without setup-specific retraining
A change of surface, lighting, or device pose often invalidates a setup-specific model. SIComp separates online geometric correction from photometric compensation in a co-adaptive design. An optical-flow module handles geometry, while the photometric network uses intensity-varying surface priors to support unseen setups without fine-tuning or retraining.
Generative photometric compensation
DiffPC explores a diffusion-based solution to the photometric component. It represents projection-induced photometric distortion as environment-dependent additive noise and generates compensation images through physically constrained denoising. The noise estimator combines photometric features and content conditions, connecting the physical projection process with perceptual image generation. Its scope is photometric compensation; the full geometric correction problem remains a separate consideration.
Projected results
Papers and implementations
End-To-End Projector Photometric Compensation
Bingyao Huang, Haibin Ling
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (oral), 2019
Paper Paper (arXiv) Project Code Supplementary
CompenNet++: End-to-End Full Projector Compensation
Bingyao Huang, Haibin Ling
IEEE International Conference on Computer Vision (ICCV), 2019
Paper Paper (arXiv) Project Code Supplementary
End-to-End Full Projector Compensation
Bingyao Huang, Tao Sun, Haibin Ling
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022
Some preliminary results appear in CVPR 2019 and ICCV 2019.
Paper Paper (arXiv) Project Code Supplementary (Google Drive)
CompenHR: Efficient Full Compensation for High-Resolution Projector
Yuxi Wang, Haibin Ling, Bingyao Huang
IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2023
Paper (arXiv) Paper Paper (PDF) Project Code Supplementary (Google Drive)
ViComp: Video Compensation for Projector-Camera Systems
Yuxi Wang, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2024
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2024
Paper Code
Real-Time High-Resolution Projector Compensation
Mingjie Chen, Bingyao Huang
IEEE International Conference on Multimedia and Expo (ICME), 2025
Paper Code
Setup-Independent Full Projector Compensation
Haibo Li, Qingyue Deng, Jijiang Li, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2026
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2026 (🏆Best Paper Award)
Paper Paper (arXiv) Project Code
DiffPC: Diffusion-Based Projector Photometric Compensation
Yuxi Wang, Haibin Ling, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2026
Also in IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2026
Paper Paper (arXiv) Code
Related research topics and projects
Differentiable ProCam scene models provide a complementary forward model of projection and capture. Calibration and active sensing establish the device geometry and measurements needed by many projection systems. Intelligent SAR uses compensation and surface models to support language-guided content generation and semantic interpretation.
Research topics: ProCam & SAR