← All projects

Research project

Projector-Based Adversarial Attacks

Study how physically projected patterns affect image classifiers, from stealthy perturbations to robustness across classifiers and viewpoints.

The problem

A projected pattern can temporarily change the appearance observed by a camera without modifying the object itself. An adversarial projection seeks to change a classifier’s prediction while keeping the physical perturbation subtle. Surface materials, illumination, device responses, and viewpoint changes make a successful digital perturbation difficult to realize physically.

Physical projected adversarial patterns and their effect on image classification.
The physical setup connects a projector input, a real surface, a camera observation, and a classifier prediction.

A progressively broader evaluation setting

  1. 1. Optimize stealthy physical projections

    Bring a learned project-and-capture model into adversarial and appearance optimization.

    SPAA (IEEE VR'22)

  2. 2. Reduce classifier dependence

    Aggregate losses from multiple classifiers and focus perturbations using attention-based gradient weighting.

    CAPAA (ICME'25)

  3. 3. Optimize across viewpoints

    Simulate multiple views and jointly optimize adversarial projections across classifiers and camera positions.

    VIPA (TVCG'26)

Stealthy physical projection

SPAA puts a learned project-and-capture approximation, PCNet, inside the optimization loop. The adversarial pattern is evaluated through a model of what the camera will observe after projection, rather than being optimized only as an image-space perturbation.

The optimization balances adversarial confidence with appearance changes using adversarial and stealthiness losses. This connects the physical projection constraints with the objective of producing subtle targeted or untargeted attacks.

Robustness across classifiers

CAPAA addresses dependence on an individual classifier. Its optimization aggregates adversarial and stealthiness gradients from multiple classifiers. Attention-based gradient weighting focuses perturbations on regions with high classification activation and supports robustness under camera-pose variation.

This broadens the evaluation setting beyond one fixed classifier: the pattern is developed against multiple decision models, while still accounting for the visibility of the physical perturbation.

Classifier-agnostic projected attack optimization and physical results.
CAPAA connects multiple classifiers with an attention-weighted optimization of the physical projection.

Robustness across viewpoints

VIPA adds explicit multi-view simulation of the projector-camera system. It aggregates simulated attack losses from different camera views and jointly optimizes the adversarial pattern across viewpoints and classifiers before physically projecting the result.

This progression changes the conditions under which a pattern is expected to remain effective: from a stealthy physical attack, to reduced classifier dependence, to jointly considering classifier and viewpoint variation. Each method has its own assumptions and evaluation, documented in the linked papers.

Multi-view projector-camera simulation and view-invariant adversarial projection.
VIPA uses a scene model to evaluate projected patterns from multiple viewpoints.

Demonstrations and evaluation

The figures show the physical projection setup and the later optimization frameworks. SPAA provides a video demonstration; CAPAA provides an implementation and dataset; VIPA provides a project website and dataset. Papers and available resources are listed below.

Papers and implementations

These papers document the methods and developments described above. Their authors, publication details, and available resources are listed together here.

SPAA: Stealthy Projector-Based Adversarial Attacks on Deep Image Classifiers
Bingyao Huang, Haibin Ling
IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2022
Paper Code Supplementary Video

CAPAA: Classifier-Agnostic Projector-Based Adversarial Attack
Zhan Li#, Mingyu Zhao#, Xin Dong, Haibin Ling, Bingyao Huang
IEEE International Conference on Multimedia and Expo (ICME), 2025
Paper Code

VIPA: View-Invariant Projector-Based Adversarial Attack
Jiyu Han, Qingyue Deng, Bingyao Huang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2026 (In press)
Also in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2026
Project

Datasets

Differentiable & 3D ProCam Systems develops the forward simulation models that connect a projector pattern to camera observations. Physics-Informed Computational Imaging includes Neural-STE, which studies passive optical information leakage through envelopes. That privacy problem complements this work, but does not belong to the active projector-attack progression.

Related research topics and projects

Research topics: Visual Privacy & Security