Development and validation of COMPASS: clinical evidence of orphan medicinal products – an assessment tool

BACKGROUND: Rare diseases are defined as life-threatening or chronically debilitating diseases with a prevalence of 50 out of 100,000 individuals or less. Orphan medicinal products (OMPs) are intended for the treatment of rare diseases. The assessment of quality of evidence in small populations is often complex. Many generic tools are unfit. Therefore, the aim of this study was to develop and validate a new tool to assess the quality of OMPs' clinical evidence (COMPASS). METHODS: Firstly, a draft version of the COMPASS tool, developed by the authors and consisting of three parts, was amended based on suggestions obtained in four rounds of expert consultation. Secondly, the tool was put through three rounds of validation. The data source was information provided on the Orphanet website and in European Public Assessment Report (EPAR) document of the European Medicines Agency. RESULTS: The first pilot round revealed a high (92.2%) inter-rater agreement for part one of the tool. After further improvements, the final inter-rater agreement was 86.4% for part two (on methodological quality) and three (on quality of reporting) of the tool. The COMPASS tool does not attempt to score or rank the quality of clinical evidence, but rather to give an outline of various, key elements with respect to quality of clinical evidence of OMP studies. CONCLUSIONS: The COMPASS tool can be applied to assess the quality of evidence of an OMP based on information in the registration dossier, for example by local reimbursement agencies, pharmacists or clinicians. In that way, the tool can contribute to making reimbursement and/or treatment decisions increasingly more founded on the principles of evidence-based decision making.