← All projects

Research project

Physics-Informed Computational Imaging

Recover structure and appearance by modeling how optical media distort the measurements reaching a camera.

The problem

Optical media can change both the geometry and appearance of an observation. Recovering a scene from these measurements requires knowing which changes arise from the medium and which arise from the underlying content. This project brings together two concrete imaging problems: inference through a paper envelope and simultaneous geometric/color rectification underwater.

Image formation through water and an envelope, alongside explicit ProCam scene parameters.
Physical image-formation models connect the optical medium, scene parameters, and camera observations.

Two complementary imaging systems

  1. 1. Model measurements through an envelope

    Combine geometric alignment, dehazing, and denoising to study physical-mail information leakage.

    Neural-STE (AAAI'21)

  2. 2. Rectify underwater geometry and color

    Model refraction, absorption, and scattering within a neural scene representation.

    NeuroPump (ACM MM'25)

Physical-mail imaging and privacy

Neural-STE (See-Through-Envelope) studies whether information about hidden content can be recovered from a camera image of an unopened envelope. Its model decomposes the mapping into perspective transformation, dehazing, and denoising, combining geometric alignment and appearance restoration in a convolutional network.

The recovered texture and structure reveal a privacy risk: concealing content from direct view does not necessarily prevent its recovery from camera measurements. The formulation also supports the design of envelopes that reduce this information leakage.

Neural-STE image-formation and restoration model for content hidden by an envelope.
The envelope-imaging model combines geometric alignment with dehazing and denoising.

Underwater geometric and color rectification

NeuroPump addresses a different medium: water changes light paths through refraction and changes appearance through absorption and scattering. Correcting only color or only geometry leaves the other source of distortion in place.

The system explicitly models these effects in a Neural Radiance Field pipeline. Self-supervised optimization estimates the scene and medium-related parameters together, enabling simultaneous geometric and color rectification. Controlling the decoupled parameters also supports novel views and optical-effect synthesis.

The underwater benchmark contains paired observations acquired with and without water, supporting evaluation of both geometric and color rectification.

Underwater reconstruction uses the optical model to separate scene appearance from refraction, absorption, and scattering.

Papers and implementations

Teaser for Modeling Deep Learning Based Privacy Attacks on Physical Mail

Modeling Deep Learning Based Privacy Attacks on Physical Mail
Bingyao Huang, Ruyi Lian, Dimitris Samaras, Haibin Ling
Proc. of AAAI Conference on Artificial Intelligence (AAAI), 2021
Paper Code Supplementary

Teaser for NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images

NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images
Yue Guo, Haoxiang Liao, Haibin Ling, Bingyao Huang
ACM Multimedia Conference (ACM MM), 2025
Paper Project Code Supplementary

Related research topics and projects

DPCS, described in Differentiable & 3D ProCam Systems, uses explicit differentiable light transport to estimate scene parameters and simulate projection. Projector-Based Adversarial Attacks studies another connection between physical image formation and visual security: manipulating camera observations with projected light.

Research topics: 3D Vision & Computational Imaging ยท Visual Privacy & Security