Validität und Reliabilität virtualisierter RTR-Messungen

Real-Time Response Measurement (RTR) is an important approach in political communication research to investigate human processing of political information. Since Reinemann et al. demonstrated the reliability and validity of RTR with physical input devices in laboratory experimental study designs in this journal 15 years ago, measurement technology has experienced an instrumental change characterized by its virtualization. At the same time, there still is little methodological knowledge available about the fundamental changes in this key technology. Therefore, this article examines the extent to which data collected in a field study using a virtualized RTR-Measurement tool meet established standards of data quality. Therefore, our first research question examines the validity of the RTR data in three ways. First, we consider construct validity. The results of our Kruskal-Wallis tests show a statistically significant differentiation of RTR rating behavior along party lines. This positive assessment is underlined by the high match score of a discriminant analysis. In a further analysis step, we implement a sparse structural equation model that assesses the relationship between party identification and RTR measurement, allowing us to draw conclusions about the construct validity of the virtualized RTR measurement. The calculated coefficients provide evidence that party identification and real-time rating are significantly and substantially associated. Although our analysis focuses merely on the partisan groups of the two candidates which parties are represented in the TV debates, in summarizing the results, we consider the question of construct validity (FF1.1) to be positively decided. As a second perspective, we investigate criterion validity. A central finding here emerges from an examination of the sparse structural equation model, at the core of which is the relationship of RTR ratings and retrospective verdicts on debate performance. Controlling for party identification, we can show that the coefficients are very strong and significant. This observation holds across all duels and candidates. We thus find a consistent pattern, which leads us to answer the question about criterion validity (FF1.2) of our virtualized RTR measure in the affirmative. As a third perspective, we focus on content validity. For this purpose, we refer to an established structural equation model that embeds the RTR signal in a complex framework of upstream and downstream variables of debate reception. Our results are broadly consistent across duels and candidates: real-time evaluations during the debate are significantly preformed by political predispositions (prior candidate evaluation). Consistent with our previous findings, perceived debate performance is largely a function of RTR ratings in this more complex model; this underscores the previously positive assessment about the criterion validity of the virtualized RTR measure. In turn, the retrospective candidate evaluations are significantly shaped by the real-time ratings and the retrospective verdicts on debate performance. Consequently, the framework used shows that the structure of debate perception in our field studies corresponds to findings known from studies in laboratory settings with physical input devices. We are therefore confident to answer the question of content validity (FF1.3) of our virtualized RTR measurement positively. Our second research question focuses on the reliability of the RTR data and examines it in two ways. First, using a resampling approach, we take a close look at the aggregated RTR time series. On the one hand, we find that virtualized RTR measures can indeed generate data whose aggregate RTR time series for two subsamples are clearly associated with each other in the resampling procedure. On the other hand, the findings are ambivalent because the existing correlations are only of moderate strength and in some cases only weak. Thus, the RTR measure seems to depend on the composition of the sample to a considerable extent. In a second analysis strategy, we check the individual RTR time series for internal consistency. Both Cronbach’s alpha and McDonald’s omega demonstrate a high degree of intercorrelation between the RTR ratings of the individual speech phases. When summarizing our findings on the aggregated and individual RTR time series, we must acknowledge limitations to our research question two (FF2): Virtualized RTR measurements can generate reliable data. However, this is not an automatism; rather, researchers must take special caution to ensure the conditions for an adequate study design in the online setting. Acknowledging that our results are subject to multiple limitations, we conclude that the virtualization of RTR-Measurement establishes a complementary approach to studies with a laboratory research design dominating the literature, making the analysis of viewers’ reactions to televised debates in natural reception situations accessible even in times of the COVID-19 pandemic.