Understanding deep features with computer-generated imagery

Mathieu Aubry and Bryan Russell

Photo is broken Photo is broken


We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses corresponding to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three CNNs trained on large image datasets: AlexNet, Places, and Oxford VGG. We observe important differences across the networks and CNN layers for different scene factors and object categories. Finally, we demonstrate that our analysis based on computer-generated imagery translates to the network representation of natural images.

Code and data

The code for the quantitative analysis is available on https://github.com/mathieuaubry/features_analysis , together with links to the rendered images of the objects used in this project. We also plan to release our renderer soon.

Supplementary material

The supplementary material for this project can be accessed here .


poster thumbnail


Article thumbnail

Understanding deep features with computer-generated imagery

M. Aubry and B. Russell

ICCV, 2015

Download pdf (ArXiv preprint) | bibtex

Spotlight video

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.