The impact of an audience response system on a summative assessment, a controlled field study

BACKGROUND: Audience response systems allow to activate the audience and to receive a direct feedback of participants during lectures. Modern systems do not require any proprietary hardware anymore. Students can directly respond on their smartphone. Several studies reported about a high level of satisfaction of students when audience response systems are used, however their impact on learning success is still unclear. METHODS: In order to evaluate the impact of an audience response system on the learning success we implemented the audience response system eduVote into a seminar series and performed a controlled crossover study on its impact on assessments. One hundred fifty-four students in nine groups were taught the same content. In four groups, eduVote was integrated for the first topic while five groups were taught this topic without the audience response systems. For a second topic, the groups were switched: Those groups who were taught before using eduVote were now taught without the audience response system and vice versa. We then analysed the impact of the audience response system on the students’ performance in a summative assessment and specifically focused on questions dealing with the topic, for which the audience response system was used during teaching. We further assessed the students’ perception on the use of eduVote using questionnaires. RESULTS: In our controlled crossover study we could not confirm an impact of the audience response system eduVote on long-term persistence i.e. the students’ performance in the summative assessment. Our evaluation revealed that students assessed the use of eduVote very positively, felt stronger engaged and better motivated to deal with the respective topics and would prefer their integration into additional courses as well. In particular we identified that students who feel uncomfortable with answering questions in front of others profit from the use of an audience response system during teaching. CONCLUSIONS: Audience response systems motivate and activate students and increase their engagement during classes. However, their impact on long-term persistence and summative assessments may be limited. Audience response systems, however, specifically allow activating students which cannot be reached by the traditional way of asking questions without such an anonymous tool.