Figure Interpretation Assessment Tool-Health (FIAT-health) 2.0: from a scoring instrument to a critical appraisal tool

BACKGROUND: Statistics are frequently used in health advocacy to attract attention, but are often misinterpreted. The Figure Interpretation Assessment Tool–Health (FIAT-Health) 1.0 was developed to support systematic assessment of the interpretation of figures on health and health care. This study aimed to test and evaluate the FIAT-Health 1.0 amongst its intended user groups, and further refine the tool based on our results. METHODS: Potential users (N = 32) were asked to assess one publicly reported figure using the FIAT-Health 1.0, and to justify their assessments and share their experience in using the FIAT-Health. In total four figures were assessed. For each figure, an expert on the specific topic (N = 4) provided a comparative assessment. The consistency of the answers was calculated, and answers to the evaluation questions were qualitatively analysed. A qualitative comparative analysis of the justifications for assessment by the experts and potential users was made. Based on the results, a new version of the FIAT-Health was developed and tested by employees (N = 27) of the National Institute for Public Health and the Environment (RIVM), and approved by the project’s advisory group. In total sixty-three participants contributed. RESULTS: Potential users using the FIAT-Health 1.0 and experts gave similar justifications for their assessments. The justifications provided by experts aligned with the items of the FIAT-Health. Seventeen out of twenty-six dichotomous questions were consistently answered by the potential users. Numerical assessment questions showed inconsistencies in how potential users responded. In the evaluation, potential users most frequently mentioned that thanks to its structured approach, the FIAT-Health contributed to their awareness of the main characteristics of the figure (n = 14), but they did find the tool complex (n = 11). The FIAT-Health 1.0 was revised from a scoring instrument into a critical appraisal tool: the FIAT-Health 2.0, which was tested and approved by employees of the RIVM and the advisory group. CONCLUSION: The tool was refined according to the results of the test and evaluation, transforming the FIAT-Health from a quantitative scoring instrument into an online qualitative appraisal tool that has the potential to aid the better interpretation and public reporting of statistics on health and healthcare. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s12874-019-0797-6) contains supplementary material, which is available to authorized users.