The problem
Optical media can change both the geometry and appearance of an observation. Recovering a scene from these measurements requires knowing which changes arise from the medium and which arise from the underlying content. This project brings together two concrete imaging problems: inference through a paper envelope and simultaneous geometric/color rectification underwater.
Two complementary imaging systems
-
1. Model measurements through an envelope
Combine geometric alignment, dehazing, and denoising to study physical-mail information leakage.
-
2. Rectify underwater geometry and color
Model refraction, absorption, and scattering within a neural scene representation.
These are separate applications of physical modeling, with different media, measurements, and tasks. Their shared approach is to incorporate the structure of image formation into inference rather than treating distortion as an unconstrained image-to-image mapping.
Physical-mail imaging and privacy
Neural-STE (See-Through-Envelope) studies whether information about hidden content can be recovered from a camera image of an unopened envelope. Its model decomposes the mapping into perspective transformation, dehazing, and denoising, combining geometric alignment and appearance restoration in a convolutional network.
The recovered texture and structure reveal a privacy risk: concealing content from direct view does not necessarily prevent its recovery from camera measurements. The formulation also supports the design of envelopes that reduce this information leakage.
Underwater geometric and color rectification
NeuroPump addresses a different medium: water changes light paths through refraction and changes appearance through absorption and scattering. Correcting only color or only geometry leaves the other source of distortion in place.
The system explicitly models these effects in a Neural Radiance Field pipeline. Self-supervised optimization estimates the scene and medium-related parameters together, enabling simultaneous geometric and color rectification. Controlling the decoupled parameters also supports novel views and optical-effect synthesis.
Results and data
Neural-STE provides an implementation, supplementary material, and a dataset for the envelope-imaging problem. NeuroPump provides an implementation and an underwater benchmark with paired observations acquired with and without water, supporting evaluation of both geometric and color rectification. Papers, code, and datasets for both systems are listed below.
Papers and implementations
These papers document the methods and developments described above. Their authors, publication details, and available resources are listed together here.
Modeling Deep Learning Based Privacy Attacks on Physical Mail
Bingyao Huang, Ruyi Lian, Dimitris Samaras, Haibin Ling
Proc. of AAAI Conference on Artificial Intelligence (AAAI), 2021
Paper Code Supplementary
NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images
Yue Guo, Haoxiang Liao, Haibin Ling, Bingyao Huang
ACM Multimedia Conference (ACM MM), 2025
Paper Project Code Supplementary
Datasets
Related work
DPCS, described in Differentiable & 3D ProCam Systems, uses explicit differentiable light transport to estimate scene parameters and simulate projection. Projector-Based Adversarial Attacks studies another connection between physical image formation and visual security: manipulating camera observations with projected light.
Related research topics and projects
Research topics: 3D Vision & Computational Imaging · Visual Privacy & Security