Can women’s reports in client exit interviews be used to measure and track progress of antenatal care services quality? Evidence from a facility assessment census in Malawi

INTRODUCTION: Unlike household surveys, client exit interviews are conducted immediately after a consultation and therefore provides an opportunity to capture routine performance and level of service quality. This study examines the validity and reliability of women’s reports on selected ANC interventions in exit interviews conducted in Malawi. METHODS: Using data from the 2013–2014 Malawi service provision facility census, we compared women’s reports in exit interviews regarding the contents of ANC received with reports obtained through direct observation by a trained healthcare professional. The validity of six indicators was tested using two measures: the area under the receiver operating characteristic curve (AUC), and the inflation factor (IF). Reliability of women’s reports was measured using the Kappa coefficient (κ) and the prevalence-adjusted bias-adjusted kappa (PABAK). Finally, we examined whether reporting reliability varied significantly by individual and facility characteristics. RESULTS: Of the six indicators, two concrete and observable measures had high reporting accuracy and met the validity criteria for both AUC ≥ 0.7 and 0.75>IF>1.25, namely whether the provider prescribed or gave malaria prophylaxis (AUC: 0.84, 95% CI: 0.83–0.86; IF: 0.96) or iron/folic tablets (AUC: (0.84 95% CI: 0.81–0.87; IF:1.00). Whereas four measures related to counselling had lower reporting accuracy: whether the provider offered counselling about nutrition in pregnancy (AUC: 0.69, 95%CI: 0.67–0.71; IF = 1.26), delivery preparation (AUC: 0.62, 95% CI: 0.60–065; IF = 0.99), pregnancy related complications (AUC: 0.59, 95%CI: 0.56–0.61; IF = 1.11), and iron/folic acid side effects (AUC:0.58, 95% CI: 0.55–0.60; IF = 1.42). Similarly, the observable measures had high reliability with both κ and PABAK values in the ranges of ≥ 0.61 and ≥ 0.80. Respondent’s age, primiparous status, number of antenatal visits, and the type of health provider increased the likelihood of reporting reliability. CONCLUSION: In order to enhance the measurement of quality of ANC services, our study emphasizes the importance of carefully considering the type of information women are asked to recall and the timing of the interviews. While household survey programmes such as the demographic health survey and multiple indicator cluster survey are commonly used as data sources for measuring intervention coverage and quality, policy makers should complement such data with more reliable sources like routine data from health information systems.