Application of MCAT questions as a testing tool and evaluation metric for knowledge graph–based reasoning systems

“Knowledge graphs” (KGs) have become a common approach for representing biomedical knowledge. In a KG, multiple biomedical data sets can be linked together as a graph representation, with nodes representing entities, such as “chemical substance” or “genes,” and edges representing predicates, such as “causes” or “treats.” Reasoning and inference algorithms can then be applied to the KG and used to generate new knowledge. We developed three KG‐based question‐answering systems as part of the Biomedical Data Translator program. These systems are typically tested and evaluated using traditional software engineering tools and approaches. In this study, we explored a team‐based approach to test and evaluate the prototype “Translator Reasoners” through the application of Medical College Admission Test (MCAT) questions. Specifically, we describe three “hackathons,” in which the developers of each of the three systems worked together with a moderator to determine whether the applications could be used to solve MCAT questions. The results demonstrate progressive improvement in system performance, with 0% (0/5) correct answers during the first hackathon, 75% (3/4) correct during the second hackathon, and 100% (5/5) correct during the final hackathon. We discuss the technical and sociologic lessons learned and conclude that MCAT questions can be applied successfully in the context of moderated hackathons to test and evaluate prototype KG‐based question‐answering systems, identify gaps in current capabilities, and improve performance. Finally, we highlight several published clinical and translational science applications of the Translator Reasoners.