Cargando…
Exploring the use of topological data analysis to automatically detect data quality faults
Data quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance functio...
Autor principal: | |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9760691/ https://www.ncbi.nlm.nih.gov/pubmed/36545477 http://dx.doi.org/10.3389/fdata.2022.931398 |
_version_ | 1784852533610545152 |
---|---|
author | Tudoreanu, M. Eduard |
author_facet | Tudoreanu, M. Eduard |
author_sort | Tudoreanu, M. Eduard |
collection | PubMed |
description | Data quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance function is used to transform each record of a dataset into an n-dimensional vector of real numbers, which effectively transforms the original data into a high-dimensional point cloud. The shape of the point cloud is then efficiently examined via topological data analysis to find high-dimensional anomalies that may signal quality issues. The specific quality faults examined in this paper are the detection of records that, while not exactly the same, refer to the same entity. Our algorithm, based on topological data analysis, provides similar accuracy for both higher and lower quality data and performs better than a baseline approach for data with poor quality. |
format | Online Article Text |
id | pubmed-9760691 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-97606912022-12-20 Exploring the use of topological data analysis to automatically detect data quality faults Tudoreanu, M. Eduard Front Big Data Big Data Data quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance function is used to transform each record of a dataset into an n-dimensional vector of real numbers, which effectively transforms the original data into a high-dimensional point cloud. The shape of the point cloud is then efficiently examined via topological data analysis to find high-dimensional anomalies that may signal quality issues. The specific quality faults examined in this paper are the detection of records that, while not exactly the same, refer to the same entity. Our algorithm, based on topological data analysis, provides similar accuracy for both higher and lower quality data and performs better than a baseline approach for data with poor quality. Frontiers Media S.A. 2022-12-05 /pmc/articles/PMC9760691/ /pubmed/36545477 http://dx.doi.org/10.3389/fdata.2022.931398 Text en Copyright © 2022 Tudoreanu. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Big Data Tudoreanu, M. Eduard Exploring the use of topological data analysis to automatically detect data quality faults |
title | Exploring the use of topological data analysis to automatically detect data quality faults |
title_full | Exploring the use of topological data analysis to automatically detect data quality faults |
title_fullStr | Exploring the use of topological data analysis to automatically detect data quality faults |
title_full_unstemmed | Exploring the use of topological data analysis to automatically detect data quality faults |
title_short | Exploring the use of topological data analysis to automatically detect data quality faults |
title_sort | exploring the use of topological data analysis to automatically detect data quality faults |
topic | Big Data |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9760691/ https://www.ncbi.nlm.nih.gov/pubmed/36545477 http://dx.doi.org/10.3389/fdata.2022.931398 |
work_keys_str_mv | AT tudoreanumeduard exploringtheuseoftopologicaldataanalysistoautomaticallydetectdataqualityfaults |