Application of data mining techniques for predicting residents' performance on pre-board examinations: A case study

CONTEXT: Predicting residents' academic performance is critical for medical educational institutions to plan strategies for improving their achievement. AIMS: This study aimed to predict the performance of residents on preboard examinations based on the results of in-training examinations (ITE) using various educational data mining (DM) techniques. SETTINGS AND DESIGN: This research was a descriptive cross-sectional pilot study conducted at Iran University of Medical Sciences, Iran. PARTICIPANTS AND METHODS: A sample of 841 residents in six specialties participating in the ITEs between 2004 and 2014 was selected through convenience sampling. Data were collected from the residency training database using a researcher-made checklist. STATISTICAL ANALYSIS: The analysis of variance was performed to compare mean scores between specialties, and multiple-regression was conducted to examine the relationship between the independent variables (ITEs scores in postgraduate 1(st) year [PGY1] to PG 3(rd) year [PGY3], sex, and type of specialty training) and the dependent variable (scores of postgraduate 4(th) year called preboard). Next, three DM algorithms, including multi-layer perceptron artificial neural network (MLP-ANN), support vector machine, and linear regression were utilized to build the prediction models of preboard examination scores. The performance of models was analyzed based on the root mean square error (RMSE) and mean absolute error (MAE). In the final step, the MLP-ANN was employed to find the association rules. Data analysis was performed in SPSS 22 and RapidMiner 7.1.001. RESULTS: The ITE scores on the PGY-2 and PGY-3 and the type of specialty training were the predictors of scores on the preboard examination (R(2) = 0.129, P < 0.01). The algorithm with the overall best results in terms of measuring error values was MLP-ANN with the condition of ten-fold cross-validation (RMSE = 0.325, MAE = 0.212). Finally, MLP-ANN was utilized to find the efficient rules. CONCLUSIONS: According to the results of the study, MLP-ANN was recognized to be useful in the evaluation of student performance on the ITEs. It is suggested that medical, educational databases be enhanced to benefit from the potential of DM approach in the identification of residents at risk, allowing instructors to offer constructive advice in a timely manner.