On the reliability of individual economic rationality measurements

A contemporary research agenda in behavioral economics and neuroeconomics aims to identify individual differences and (neuro)psychological correlates of rationality. This research has been widely received in important interdisciplinary and field outlets. However, the psychometric reliability of such measurements of rationality has been presumed without enough methodological scrutiny. Drawing from multiple original and published datasets (in total over 1,600 participants), we unequivocally show that contemporary measurements of rationality have moderate to poor reliability according to common standards. Further analyses of the variance components, as well as a allowing participants to revise previous choices, suggest that this is driven by low between-subject variance rather than high measurement error. As has been argued previously for other behavioral measurements, this poses a challenge to the predominant correlational research designs and the search for sociodemographic or neural predictors. While our results draw a sobering picture of the prospects of contemporary measurements of rationality, they are not necessarily surprising from a theoretical perspective, which we outline in our discussion.