Segmentation into layers: Why humans don’t try to pick up shadows

In order to determine the 3D shape of an object it is first necessary to determine which parts of the scene belong to the object and which are part of the background: figure-ground segmentation. One problem with segmenting natural images is that luminance is ambiguous, it can change for a variety of reasons including, critically, changes in both surface reflectance and the illumination field. It may therefore be beneficial to segment the image into layers representing intrinsic characteristics of the scene. Such characteristic images might include a reflectance map of the scene and the illumination field in which the scene is bathed. There is some evidence that humans can divide the visual input into such layers. Intuitively, we don’t try to avoid shadows, we don’t treat them as objects, and we don’t incorporate attached shadows into the body of an object. The reflectance map may form the basis of figure-ground segmentation, object recognition and object-level shape processing. The illumination field might yield information about shadows, surface undulations and lighting direction. I will discuss recent evidence supporting the notion that humans separate shadows and illumination from reflectance changes early in the processing stream and compare this with analogous machine vision methods for intrinsic image extraction.