Evaluating association between linguistic characteristics of abstracts and risk of bias: Case of Japanese randomized controlled trials

Despite the ongoing growth in the number of published randomized controlled trials (RCTs) and increased quality assessment of RCTs, the association between the quality and characteristics in the text has not been sufficiently studied. We are interested in a specific question: what kind of sentences is a good indicator of high quality RCTs? To help researchers to efficiently screen articles worth reading, this study aims 1) to quantify the linguistic features of articles and 2) to build a document assessment model to evaluate quality of RCTs using only the abstract. All RCTs that were conducted in Japan in 2010 as original articles were included in the analysis. Data were independently assessed by two reviewers using a risk-of-bias tool. Three aspects of linguistic style were quantitatively measured, and a document model was constructed to evaluate the RCTs. A total of 302 RCTs were selected for quality assessment. Of these, 255 articles were assessed as high quality and 47 as low quality. High-quality articles tended to use longer words than low-quality articles (p = 0.048), however sentences were generally shorter (p = 0.004). Further, high-quality articles included a larger proportion of noun phrases (p = 0.026) but a smaller proportion of verb phrases (p = 0.041). The optimal number of topics to assess the quality of articles was four, while two topics had a significant association with quality. Despite a number of articles published about RCTs in Japan, significant differences exist in several textual features between high- and low-quality RCTs. Instead of the risk-of-bias tool, these results can be used as the new criteria to rapidly screen valuable articles and it also revealed quality control of RCT articles is urgently needed in Japan.