Evaluating the Quality of Randomized Controlled Trials that Examine the Efficacy of Natural Health Products: A Systematic Review of Critical Appraisal Instruments

The purpose of this project was to conduct a systematic review to identify instruments designed to evaluate the quality of randomized controlled trials (RCTs) of natural health products (NHPs). Instruments were examined for inclusion of items assessing methods, identity and content of the NHP, generalizability of results and instructions for use. Online databases, websites, textbooks and reference lists were searched to identify instruments. Relevance assessment and data extraction of articles were completed by two investigators and disagreements were settled by the third investigator. Data were analyzed using descriptive statistics. Of the 4442 citations identified, 29 were potentially relevant with 16 meeting the criteria for inclusion. None of the instruments stated they were validated; content in the four areas of interest varied considerably. The most common items included randomization sequence generation (100%), blinding (100%), allocation concealment (75%) and participant flow (75%). Only nine of the NHP instruments included at least one item to appraise the specific content of the NHP. The CONSORT Statement for Herbal Interventions most closely addressed the four areas of interest; however, this instrument was specific for herbs. There is a need for the development of a validated instrument for assessment of the quality of RCTs that would be useful for herbs as well as other NHPs.