Gray Literature in Evaluating Effectiveness in Digital Health and Health and Welfare Technology: A Source Worth Considering

BACKGROUND: The need to assess the effectiveness and value of interventions involving digital health and health and welfare technologies is becoming increasingly important due to the rapidly growing development of these technologies and their areas of application. Systematic reviews of scientific literature are a mainstay of such assessment, but publications outside the realm of traditional scientific bibliographic databases—known as gray literature—are often not included. This is a disadvantage, particularly apparent in the health and welfare technology (HWT) domain. OBJECTIVE: The aim of this article is to investigate the significance of gray literature in digital health and HWT when reviewing literature. As an example, the impact of including gray literature to the result of two systematic reviews in HWT is examined. METHODS: In this paper, we identify, discuss, and suggest methods for including gray literature sources when evaluating effectiveness and appropriateness for different review types related to HWT. The analysis also includes established sources, search strategies, documentation, and reporting of searches, as well as bias and credibility assessment. The differences in comparison to scientific bibliographic databases are elucidated. We describe the results, challenges, and benefits of including gray literature in 2 examples of systematic reviews of HWT. RESULTS: In the 2 systematic reviews described in this paper, most included studies came from context-specific gray literature sources. Gray literature contributed to the overall result of the reviews and corresponded well with the reviews’ aims. The assessed risk of bias of the included studies derived from gray literature was similar to the included studies from other types of sources. However, because of less standardized publication formats, assessing and extracting data from gray literature studies were more time-consuming and compiling statistical results was not possible. The search process for gray literature required more time and the reproducibility of gray literature searches were less certain due to more unstable publication platforms. CONCLUSIONS: Gray literature is particularly relevant for digital health and HWT but searches need to be conducted systematically and reported transparently. This way gray literature can broaden the range of studies, highlight context specificity, and decrease the publication bias of reviews of effectiveness of HWT. Thus, researchers conducting systematic reviews related to HWT should consider including gray literature based on a systematic approach.