Concordance of multiple informant assessment of school-based social skills intervention and association with child outcomes: Results from a randomized trial

BACKGROUND: Fidelity, or the degree to which an intervention is implemented as designed, is essential for effective implementation. There has been a growing emphasis on assessing fidelity of evidence-based practices for autistic children in schools. Fidelity measurement should be multidimensional and focus on core intervention components and assess their link with program outcomes. This study evaluated the relation between intervention fidelity ratings from multiple sources, tested the relation between fidelity ratings and child outcomes, and determined the relations between core intervention components and child outcomes in a study of an evidence-based psychosocial intervention designed to promote inclusion of autistic children at school, Remaking Recess. METHOD: This study extends from a larger randomized controlled trial examining the effect of implementation support on Remaking Recess fidelity and child outcomes. Schools were randomized to receive the intervention or the intervention plus implementation support. Observers, intervention coaches, and school personnel completed fidelity measures to rate completion and quality of intervention delivery. A measure of peer engagement served as the child outcome. Pearson correlation coefficients were calculated to determine concordance between raters. Two sets of hierarchical linear models were conducted using fidelity indices as predictors of peer engagement. RESULTS: Coach- and self-rated completion and quality scores, observer- and self-rated quality scores, and observer- and coach-rated quality fidelity scores were significantly correlated. Higher observer-rated completion and quality fidelity scores were predictors of higher peer engagement scores. No single intervention component emerged as a significant predictor of peer engagement. CONCLUSIONS: This study demonstrates the importance of using a multidimensional approach for measuring fidelity, testing the link between fidelity and child outcomes, and examining how core intervention components may be associated with child outcomes. Future research should clarify how to improve multi-informant reports to provide “good enough” ratings of fidelity that provide meaningful information about outcomes in community settings. PLAIN LANGUAGE SUMMARY: Fidelity is defined as how closely an intervention is administered in the way the creators intended. Fidelity is important because it allows researchers to determine what exactly is leading to changes. In recent years, there has been an interest in examining fidelity of interventions for autistic children who receive services in school. This study looked at the relationship between fidelity ratings from multiple individuals, the relationship between fidelity and child outcomes, and the relationship between individual intervention component and child changes in a study of Remaking Recess, an intervention for autistic children at school. Schools were randomly selected to receive the intervention only or the intervention plus implementation support from the research team. Observers, intervention coaches, and individuals delivering the intervention themselves completed fidelity measures. Child engagement with peers was measured before and after the intervention. Several measures of self-, coach-, and observer-report fidelity were associated with each other. Higher observer-reported fidelity was associated with higher child peer engagement scores. No single intervention step was linked to child peer engagement and both treatment groups had similar outcomes in terms of fidelity. This study shows the importance of having multiple raters assess different parts of intervention fidelity, looking at the link between fidelity and child outcomes, and seeing how individual intervention steps may be related to outcomes. Future research should aim to find out which types of fidelity ratings are “good enough” to lead to positive changes following treatment so that those aspects can be used and targeted in the future.