How to improve the design of experimental studies in computing education: Evidence from the international assessments

Cluster randomized trials are frequently used in educational research for methodological reasons. This study aims to improve the efficiency of cluster randomized trials on computer/information literacy and computational thinking. The study employs a two-level hierarchical linear model to estimate (i) intraclass correlation coefficients, (ii) the amount of explained variances given selected predictors, and (iii) minimum detectable effect sizes given the set of plausible scenarios. Two data cycles from the International Computer and Information Study were used. The covariates at the student level are gender, interest in ICT, parents’ highest education level, ICT self-efficacy, and experience with computers. The covariates at school/teacher level are teacher’s ICT use, ratio of school size to the number of computers for student use, availability of ICT resources at school, approximate teacher age, and ICT self-efficacy. Findings showed that the most precise effect could be measured when student and teacher/school covariates are both adopted. Lastly, it was revealed that increasing the number of schools is effective to get the most precise effect.