The Dutch COVID-19 Notification App: Lessons Learned From a Mixed Methods Evaluation Among End Users and Contact-Tracing Employees

BACKGROUND: The Dutch CoronaMelder (CM) app is the official Dutch contact-tracing app (CTA). It has been used to contain the spread of the SARS-CoV-2 in the Netherlands. It allows its users and those of connected apps to anonymously exchange warnings about potentially high-risk contacts with individuals infected with the SARS-CoV-2. OBJECTIVE: The goal of this mixed methods study is to understand the use of CTA in the pandemic and its integration into the Municipal Health Services (MHS) efforts of containment through contact tracing. Moreover, the study aims to investigate both the motivations and user experience–related factors concerning adherence to quarantine and isolation measures. METHODS: A topic analysis of 56 emails and a web-based survey of 1937 adults from the Netherlands, combined with a series of 48 in-depth interviews with end users of the app and 14 employees of the Dutch MHS involved in contact tracing, were conducted. Mirroring sessions were held (n=2) with representatives from the development (n=2) and communication teams (n=2) responsible for the creation and implementation of the CM app. RESULTS: Topic analysis and interviews identified procedural and technical issues in the use of the CTA. Procedural issues included the lack of training of MHS employees in the use of CTAs. Technical issues identified for the end users included the inability to send notifications without phone contact with the MHS, unwarranted notifications, and nightly notifications. Together, these issues undermined confidence in and satisfaction with the app’s use. The interviews offered a deeper understanding of the various factors at play and their effects on users; for example, the mixed experiences of the app’s users, the end user’s own fears, and uncertainties concerning the SARS-CoV-2; problematic infrastructure at the time of the app’s implementation on the side of the health services; the effects of the society-wide efforts in containment of the SARS-CoV-2 on the CM app’s perception, resulting in further doubts concerning the app’s effectiveness among MHS workers and citizens; and problems with adherence to behavioral measures propagated by the app because of the lack of confidence in the app and uncertainty concerning the execution of the behavioral measures. All findings were evaluated with the app’s creators and have since contributed to improvements. CONCLUSIONS: Although most participants perceived the app positively, procedural and technical issues identified in this study limited satisfaction and confidence in the CM app and affected its adoption and long-term use. Moreover, these same issues negatively affected the CM app’s effectiveness in improving compliance with behavioral measures aimed at reducing the spread of the SARS-CoV-2. This study offers lessons learned for future eHealth interventions in pandemics. Lessons that can aid in more effective design, implementation, and communication for more effective and readily adoptable eHealth applications.