Discrimination of the hierarchical structure of cortical layers in 2-photon microscopy data by combined unsupervised and supervised machine learning

The laminar organization of the cerebral cortex is a fundamental characteristic of the brain, with essential implications for cortical function. Due to the rapidly growing amount of high-resolution brain imaging data, a great demand arises for automated and flexible methods for discriminating the laminar texture of the cortex. Here, we propose a combined approach of unsupervised and supervised machine learning to discriminate the hierarchical cortical laminar organization in high-resolution 2-photon microscopic neural image data of mouse brain without observer bias, that is, without the prerequisite of manually labeled training data. For local cortical foci, we modify an unsupervised clustering approach to identify and represent the laminar cortical structure. Subsequently, supervised machine learning is applied to transfer the resulting layer labels across different locations and image data, to ensure the existence of a consistent layer label system. By using neurobiologically meaningful features, the discrimination results are shown to be consistent with the layer classification of the classical Brodmann scheme, and provide additional insight into the structure of the cerebral cortex and its hierarchical organization. Thus, our work paves a new way for studying the anatomical organization of the cerebral cortex, and potentially its functional organization.