Physically Grounded Scene Interpretation

How can we interpret images in terms of interacting objects and surfaces? We aim to better localize and recognize objects and to interpret scenes using estimates of the physical scene space.


NYU Depth Dataset v2: dataset and and RGBD segmentation and support inference code from ECCV'12 paper

Shadow Detection and Removal: code and dataset

Generic Object Proposals: including object regions for BSDS dataset and object proposal code

Photo Clip Art: demo and video

Putting Objects in Perspective: dataset and code

Publications and Projects

Category-Independent Object Proposals with Diverse Ranking
accepted to PAMI. [pdf] [project]

Indoor Segmentation and Support Inference from RGBD Images
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus
ECCV, 2012. [pdf] [project] [slides]

Single-Image Shadow Detection and Removal using Paired Regions
R. Guo and Q. Dai and D. Hoiem
CVPR, 2011. [pdf] [project]

Category Independent Object Proposals
I. Endres and D. Hoiem
ECCV 2010. [pdf] [project]

Thinking Inside the Box: Using Appearance Models and Context Based on Room Geometry
V. Hedau, D. Hoiem, and D.A. Forsyth
ECCV 2010. [pdf]

An Empirical Study of Context in Object Detection
S.K. Divvala, D. Hoiem, J.H. Hays, A.A. Efros, and M. Hebert
CVPR 2009. [pdf]

Closing the Loop on Scene Interpretation
D. Hoiem, A.A. Efros, and M. Hebert
CVPR 2008. [pdf]

Putting Objects in Perspective
D. Hoiem, A.A. Efros, and M. Hebert
IJCV (80), No. 1, October 2008. [pdf]
The final publication is available at

Seeing the World Behind the Image: Spatial Layout for 3D Scene Understanding
D. Hoiem
Doctoral Dissertation, CMU-RI-TR-07-28, Robotics Institute, Carnegie Mellon University, August 2007. [pdf]
CMU School of Computer Science Distinguished Dissertation Award
ACM Doctoral Dissertation Honorable Mention

Photo Clip Art
J-F. Lalonde, D. Hoiem, A.A. Efros, J. Winn, C. Rother and A. Criminisi
ACM SIGGRAPH 2007. [pdf] [project ]

Putting Objects in Perspective
D. Hoiem, A.A. Efros, and M. Hebert
CVPR 2006. Best Paper Award [pdf] [project]

Opportunistic use of vision to push back the path-planning horizon
B. Nabbe, D. Hoiem, A.A. Efros, and M. Hebert
IROS 2006. [pdf]

Funding Sources

NSF Award 0916014 for "Exploiting Geometric and Illumination Context in Indoor Scenes" (2009-2012)

NSF Award 0904209 for "Physically Grounded Object Recognition" (2009-2013)

Gift from Microsoft