Learning to see stuff

Materials with complex appearances, like textiles and foodstuffs, pose challenges for conventional theories of vision. But recent advances in unsupervised deep learning provide a framework for explaining how we learn to see them. We suggest that perception does not involve estimating physical quantities like reflectance or lighting. Instead, representations emerge from learning to encode and predict the visual input as efficiently and accurately as possible. Neural networks can be trained to compress natural images or to predict frames in movies without ‘ground truth’ data about the outside world. Yet, to succeed, such systems may automatically discover how to disentangle distal causal factors. Such ‘statistical appearance models’ potentially provide a coherent explanation of both failures and successes in perception.