Toward a statistical validation of brain signatures as robust measures of behavioral substrates

The “brain signature of cognition” concept has garnered interest as a data‐driven, exploratory approach to better understand key brain regions involved in specific cognitive functions, with the potential to maximally characterize brain substrates of behavioral outcomes. Previously we presented a method for computing signatures of episodic memory. However, to be a robust brain measure, the signature approach requires a rigorous validation of model performance across a variety of cohorts. Here we report validation results and provide an example of extending it to a second behavioral domain. In each of two discovery data cohorts, we derived regional brain gray matter thickness associations for two domains: neuropsychological and everyday cognition memory. We computed regional association to outcome in 40 randomly selected discovery subsets of size 400 in each cohort. We generated spatial overlap frequency maps and defined high‐frequency regions as “consensus” signature masks. Using separate validation datasets, we evaluated replicability of cohort‐based consensus model fits and explanatory power by comparing signature model fits with each other and with competing theory‐based models. Spatial replications produced convergent consensus signature regions. Consensus signature model fits were highly correlated in 50 random subsets of each validation cohort, indicating high replicability. In comparisons over each full cohort, signature models outperformed other models. In this validation study, we produced signature models that replicated model fits to outcome and outperformed other commonly used measures. Signatures in two memory domains suggested strongly shared brain substrates. Robust brain signatures may therefore be achievable, yielding reliable and useful measures for modeling substrates of behavioral domains.