Beyond ℓ(1) sparse coding in V1

Growing evidence indicates that only a sparse subset from a pool of sensory neurons is active for the encoding of visual stimuli at any instant in time. Traditionally, to replicate such biological sparsity, generative models have been using the ℓ(1) norm as a penalty due to its convexity, which makes it amenable to fast and simple algorithmic solvers. In this work, we use biological vision as a test-bed and show that the soft thresholding operation associated to the use of the ℓ(1) norm is highly suboptimal compared to other functions suited to approximating ℓ(p) with 0 ≤ p < 1 (including recently proposed continuous exact relaxations), in terms of performance. We show that ℓ(1) sparsity employs a pool with more neurons, i.e. has a higher degree of overcompleteness, in order to maintain the same reconstruction error as the other methods considered. More specifically, at the same sparsity level, the thresholding algorithm using the ℓ(1) norm as a penalty requires a dictionary of ten times more units compared to the proposed approach, where a non-convex continuous relaxation of the ℓ(0) pseudo-norm is used, to reconstruct the external stimulus equally well. At a fixed sparsity level, both ℓ(0)- and ℓ(1)-based regularization develop units with receptive field (RF) shapes similar to biological neurons in V1 (and a subset of neurons in V2), but ℓ(0)-based regularization shows approximately five times better reconstruction of the stimulus. Our results in conjunction with recent metabolic findings indicate that for V1 to operate efficiently it should follow a coding regime which uses a regularization that is closer to the ℓ(0) pseudo-norm rather than the ℓ(1) one, and suggests a similar mode of operation for the sensory cortex in general.