Empirical Study of Overfitting in Deep Learning for Predicting Breast Cancer Metastasis

SIMPLE SUMMARY: It is important to be able to effectively predict the likelihood of breast cancer metastasis to potentially help make treatment plans for a patient. We developed a type of deep learning models called feedforward neural network (FNN) models to predict breast cancer metastasis using clinical data. We found that overfitting can affect the prediction performance negatively, and overfitting and model performance can be greatly affected by hyperparameter settings. In this research, we conducted grid search experiments to study how each of the 11 hyperparameters of our FNN models is related to overfitting and model performance. Our experiment results show that the top five hyperparameters that have a significant impact on overfitting are iteration-based decay, learning rate, batch size, L2, and L1. The experiment results illustrate that different hyperparameters have a different impact on overfitting, and hyperparameter tuning with grid search can help improve the prediction performance of the FNN models. ABSTRACT: Overfitting may affect the accuracy of predicting future data because of weakened generalization. In this research, we used an electronic health records (EHR) dataset concerning breast cancer metastasis to study the overfitting of deep feedforward neural networks (FNNs) prediction models. We studied how each hyperparameter and some of the interesting pairs of hyperparameters were interacting to influence the model performance and overfitting. The 11 hyperparameters we studied were activate function, weight initializer, number of hidden layers, learning rate, momentum, decay, dropout rate, batch size, epochs, L1, and L2. Our results show that most of the single hyperparameters are either negatively or positively corrected with model prediction performance and overfitting. In particular, we found that overfitting overall tends to negatively correlate with learning rate, decay, batch size, and L2, but tends to positively correlate with momentum, epochs, and L1. According to our results, learning rate, decay, and batch size may have a more significant impact on both overfitting and prediction performance than most of the other hyperparameters, including L1, L2, and dropout rate, which were designed for minimizing overfitting. We also find some interesting interacting pairs of hyperparameters such as learning rate and momentum, learning rate and decay, and batch size and epochs.