Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review

BACKGROUND: A relatively novel method of appraisal, methodological reviews (MRs) are used to synthesize information on the methods used in health research. There are currently no guidelines available to inform the reporting of MRs. OBJECTIVES: This pilot review aimed to determine the feasibility of a full review and the need for reporting guidance for methodological reviews. METHODS: Search strategy: We conducted a search of PubMed, restricted to 2017 to include the most recently published studies, using different search terms often used to describe methodological reviews: “literature survey” OR “meta-epidemiologic* review” OR “meta-epidemiologic* survey” OR “methodologic* review” OR “methodologic* survey” OR “systematic survey.” Data extraction: Study characteristics including country, nomenclature, number of included studies, search strategy, a priori protocol use, and sampling methods were extracted in duplicate and summarized. Outcomes: Primary feasibility outcomes were the sensitivity and specificity of the search terms (criteria for success of feasibility set at sensitivity and specificity of ≥ 70%). Analysis: The estimates are reported as a point estimate (95% confidence interval). RESULTS: Two hundred thirty-six articles were retrieved and 31 were included in the final analysis. The most accurate search term was “meta-epidemiological” (sensitivity [Sn] 48.39; 95% CI 31.97–65.16; specificity [Sp] 97.56; 94.42–98.95). The majority of studies were published by authors from Canada (n = 12, 38.7%), and Japan and USA (n = 4, 12.9% each). The median (interquartile range [IQR]) number of included studies in the MRs was 77 (13–1127). Reporting of a search strategy was done in most studies (n = 23, 74.2%). The use of a pre-published protocol (n = 7, 22.6%) or a justifiable sampling method (n = 5, 16.1%) occurred rarely. CONCLUSIONS: Using the MR nomenclature identified, it is feasible to build a comprehensive search strategy and conduct a full review. Given the variation in reporting practices and nomenclature attributed to MRs, there is a need for guidance on standardized and transparent reporting of MRs. Future guideline development would likely include stakeholders from Canada, USA, and Japan.