Cargando…
The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity
In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is si...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7418972/ https://www.ncbi.nlm.nih.gov/pubmed/32780738 http://dx.doi.org/10.1371/journal.pone.0237009 |
_version_ | 1783569789999906816 |
---|---|
author | Schat, Evelien van de Schoot, Rens Kouw, Wouter M. Veen, Duco Mendrik, Adriënne M. |
author_facet | Schat, Evelien van de Schoot, Rens Kouw, Wouter M. Veen, Duco Mendrik, Adriënne M. |
author_sort | Schat, Evelien |
collection | PubMed |
description | In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is similar to new unseen data one wishes to apply it to. It is often unknown in advance how an algorithm will perform on new unseen data, being a crucial reason for not deploying an algorithm at all. Therefore, tools are needed to measure the similarity of data sets. In this paper, we propose the Data Representativeness Criterion (DRC) to determine how representative a training data set is of a new unseen data set. We present a proof of principle, to see whether the DRC can quantify the similarity of data sets and whether the DRC relates to the performance of a supervised classification algorithm. We compared a number of magnetic resonance imaging (MRI) data sets, ranging from subtle to severe difference is acquisition parameters. Results indicate that, based on the similarity of data sets, the DRC is able to give an indication as to when the performance of a supervised classifier decreases. The strictness of the DRC can be set by the user, depending on what one considers to be an acceptable underperformance. |
format | Online Article Text |
id | pubmed-7418972 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-74189722020-08-19 The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity Schat, Evelien van de Schoot, Rens Kouw, Wouter M. Veen, Duco Mendrik, Adriënne M. PLoS One Research Article In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is similar to new unseen data one wishes to apply it to. It is often unknown in advance how an algorithm will perform on new unseen data, being a crucial reason for not deploying an algorithm at all. Therefore, tools are needed to measure the similarity of data sets. In this paper, we propose the Data Representativeness Criterion (DRC) to determine how representative a training data set is of a new unseen data set. We present a proof of principle, to see whether the DRC can quantify the similarity of data sets and whether the DRC relates to the performance of a supervised classification algorithm. We compared a number of magnetic resonance imaging (MRI) data sets, ranging from subtle to severe difference is acquisition parameters. Results indicate that, based on the similarity of data sets, the DRC is able to give an indication as to when the performance of a supervised classifier decreases. The strictness of the DRC can be set by the user, depending on what one considers to be an acceptable underperformance. Public Library of Science 2020-08-11 /pmc/articles/PMC7418972/ /pubmed/32780738 http://dx.doi.org/10.1371/journal.pone.0237009 Text en © 2020 Schat et al http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Schat, Evelien van de Schoot, Rens Kouw, Wouter M. Veen, Duco Mendrik, Adriënne M. The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title | The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title_full | The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title_fullStr | The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title_full_unstemmed | The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title_short | The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity |
title_sort | data representativeness criterion: predicting the performance of supervised classification based on data set similarity |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7418972/ https://www.ncbi.nlm.nih.gov/pubmed/32780738 http://dx.doi.org/10.1371/journal.pone.0237009 |
work_keys_str_mv | AT schatevelien thedatarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT vandeschootrens thedatarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT kouwwouterm thedatarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT veenduco thedatarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT mendrikadriennem thedatarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT schatevelien datarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT vandeschootrens datarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT kouwwouterm datarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT veenduco datarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity AT mendrikadriennem datarepresentativenesscriterionpredictingtheperformanceofsupervisedclassificationbasedondatasetsimilarity |