Cargando…
Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study
BACKGROUND: Accurate solutions for the estimation of physical activity and energy expenditure at scale are needed for a range of medical and health research fields. Machine learning techniques show promise in research-grade accelerometers, and some evidence indicates that these techniques can be app...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
JMIR Publications
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8374660/ https://www.ncbi.nlm.nih.gov/pubmed/34346890 http://dx.doi.org/10.2196/23938 |
_version_ | 1783740163972661248 |
---|---|
author | O'Driscoll, Ruairi Turicchi, Jake Hopkins, Mark Duarte, Cristiana Horgan, Graham W Finlayson, Graham Stubbs, R James |
author_facet | O'Driscoll, Ruairi Turicchi, Jake Hopkins, Mark Duarte, Cristiana Horgan, Graham W Finlayson, Graham Stubbs, R James |
author_sort | O'Driscoll, Ruairi |
collection | PubMed |
description | BACKGROUND: Accurate solutions for the estimation of physical activity and energy expenditure at scale are needed for a range of medical and health research fields. Machine learning techniques show promise in research-grade accelerometers, and some evidence indicates that these techniques can be applied to more scalable commercial devices. OBJECTIVE: This study aims to test the validity and out-of-sample generalizability of algorithms for the prediction of energy expenditure in several wearables (ie, Fitbit Charge 2, ActiGraph GT3-x, SenseWear Armband Mini, and Polar H7) using two laboratory data sets comprising different activities. METHODS: Two laboratory studies (study 1: n=59, age 44.4 years, weight 75.7 kg; study 2: n=30, age=31.9 years, weight=70.6 kg), in which adult participants performed a sequential lab-based activity protocol consisting of resting, household, ambulatory, and nonambulatory tasks, were combined in this study. In both studies, accelerometer and physiological data were collected from the wearables alongside energy expenditure using indirect calorimetry. Three regression algorithms were used to predict metabolic equivalents (METs; ie, random forest, gradient boosting, and neural networks), and five classification algorithms (ie, k-nearest neighbor, support vector machine, random forest, gradient boosting, and neural networks) were used for physical activity intensity classification as sedentary, light, or moderate to vigorous. Algorithms were evaluated using leave-one-subject-out cross-validations and out-of-sample validations. RESULTS: The root mean square error (RMSE) was lowest for gradient boosting applied to SenseWear and Polar H7 data (0.91 METs), and in the classification task, gradient boost applied to SenseWear and Polar H7 was the most accurate (85.5%). Fitbit models achieved an RMSE of 1.36 METs and 78.2% accuracy for classification. Errors tended to increase in out-of-sample validations with the SenseWear neural network achieving RMSE values of 1.22 METs in the regression tasks and the SenseWear gradient boost and random forest achieving an accuracy of 80% in classification tasks. CONCLUSIONS: Algorithms trained on combined data sets demonstrated high predictive accuracy, with a tendency for superior performance of random forests and gradient boosting for most but not all wearable devices. Predictions were poorer in the between-study validations, which creates uncertainty regarding the generalizability of the tested algorithms. |
format | Online Article Text |
id | pubmed-8374660 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | JMIR Publications |
record_format | MEDLINE/PubMed |
spelling | pubmed-83746602021-08-24 Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study O'Driscoll, Ruairi Turicchi, Jake Hopkins, Mark Duarte, Cristiana Horgan, Graham W Finlayson, Graham Stubbs, R James JMIR Mhealth Uhealth Original Paper BACKGROUND: Accurate solutions for the estimation of physical activity and energy expenditure at scale are needed for a range of medical and health research fields. Machine learning techniques show promise in research-grade accelerometers, and some evidence indicates that these techniques can be applied to more scalable commercial devices. OBJECTIVE: This study aims to test the validity and out-of-sample generalizability of algorithms for the prediction of energy expenditure in several wearables (ie, Fitbit Charge 2, ActiGraph GT3-x, SenseWear Armband Mini, and Polar H7) using two laboratory data sets comprising different activities. METHODS: Two laboratory studies (study 1: n=59, age 44.4 years, weight 75.7 kg; study 2: n=30, age=31.9 years, weight=70.6 kg), in which adult participants performed a sequential lab-based activity protocol consisting of resting, household, ambulatory, and nonambulatory tasks, were combined in this study. In both studies, accelerometer and physiological data were collected from the wearables alongside energy expenditure using indirect calorimetry. Three regression algorithms were used to predict metabolic equivalents (METs; ie, random forest, gradient boosting, and neural networks), and five classification algorithms (ie, k-nearest neighbor, support vector machine, random forest, gradient boosting, and neural networks) were used for physical activity intensity classification as sedentary, light, or moderate to vigorous. Algorithms were evaluated using leave-one-subject-out cross-validations and out-of-sample validations. RESULTS: The root mean square error (RMSE) was lowest for gradient boosting applied to SenseWear and Polar H7 data (0.91 METs), and in the classification task, gradient boost applied to SenseWear and Polar H7 was the most accurate (85.5%). Fitbit models achieved an RMSE of 1.36 METs and 78.2% accuracy for classification. Errors tended to increase in out-of-sample validations with the SenseWear neural network achieving RMSE values of 1.22 METs in the regression tasks and the SenseWear gradient boost and random forest achieving an accuracy of 80% in classification tasks. CONCLUSIONS: Algorithms trained on combined data sets demonstrated high predictive accuracy, with a tendency for superior performance of random forests and gradient boosting for most but not all wearable devices. Predictions were poorer in the between-study validations, which creates uncertainty regarding the generalizability of the tested algorithms. JMIR Publications 2021-08-04 /pmc/articles/PMC8374660/ /pubmed/34346890 http://dx.doi.org/10.2196/23938 Text en ©Ruairi O'Driscoll, Jake Turicchi, Mark Hopkins, Cristiana Duarte, Graham W Horgan, Graham Finlayson, R James Stubbs. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 04.08.2021. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included. |
spellingShingle | Original Paper O'Driscoll, Ruairi Turicchi, Jake Hopkins, Mark Duarte, Cristiana Horgan, Graham W Finlayson, Graham Stubbs, R James Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title | Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title_full | Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title_fullStr | Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title_full_unstemmed | Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title_short | Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study |
title_sort | comparison of the validity and generalizability of machine learning algorithms for the prediction of energy expenditure: validation study |
topic | Original Paper |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8374660/ https://www.ncbi.nlm.nih.gov/pubmed/34346890 http://dx.doi.org/10.2196/23938 |
work_keys_str_mv | AT odriscollruairi comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT turicchijake comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT hopkinsmark comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT duartecristiana comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT horgangrahamw comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT finlaysongraham comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy AT stubbsrjames comparisonofthevalidityandgeneralizabilityofmachinelearningalgorithmsforthepredictionofenergyexpenditurevalidationstudy |