Emergence of Emotion Selectivity in A Deep Neural Network Trained to Recognize Visual Objects

Visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are innate to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining a convolutional neural network (CNN) model of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that (1) in all layers of the CNN model, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and (2) lesioning these neurons by setting their output to 0 or enhancing these neurons by increasing their gain lead to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the innate ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.