Cargando…
NFAD: fixing anomaly detection using normalizing flows
Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-cla...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
PeerJ Inc.
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8627226/ https://www.ncbi.nlm.nih.gov/pubmed/34901422 http://dx.doi.org/10.7717/peerj-cs.757 |
_version_ | 1784606810882179072 |
---|---|
author | Ryzhikov, Artem Borisyak, Maxim Ustyuzhanin, Andrey Derkach, Denis |
author_facet | Ryzhikov, Artem Borisyak, Maxim Ustyuzhanin, Andrey Derkach, Denis |
author_sort | Ryzhikov, Artem |
collection | PubMed |
description | Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-class SVM and Robust Auto-Encoder, are one-class classification methods, i.e., focus on separating normal data from the rest of the space. Such methods are based on the assumption of separability of normal and anomalous classes, and subsequently do not take into account any available samples of anomalies. Nonetheless, in practical settings, some anomalous samples are often available; however, usually in amounts far lower than required for a balanced classification task, and the separability assumption might not always hold. This leads to an important task—incorporating known anomalous samples into training procedures of anomaly detection models. In this work, we propose a novel model-agnostic training procedure to address this task. We reformulate one-class classification as a binary classification problem with normal data being distinguished from pseudo-anomalous samples. The pseudo-anomalous samples are drawn from low-density regions of a normalizing flow model by feeding tails of the latent distribution into the model. Such an approach allows to easily include known anomalies into the training process of an arbitrary classifier. We demonstrate that our approach shows comparable performance on one-class problems, and, most importantly, achieves comparable or superior results on tasks with variable amounts of known anomalies. |
format | Online Article Text |
id | pubmed-8627226 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | PeerJ Inc. |
record_format | MEDLINE/PubMed |
spelling | pubmed-86272262021-12-10 NFAD: fixing anomaly detection using normalizing flows Ryzhikov, Artem Borisyak, Maxim Ustyuzhanin, Andrey Derkach, Denis PeerJ Comput Sci Artificial Intelligence Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-class SVM and Robust Auto-Encoder, are one-class classification methods, i.e., focus on separating normal data from the rest of the space. Such methods are based on the assumption of separability of normal and anomalous classes, and subsequently do not take into account any available samples of anomalies. Nonetheless, in practical settings, some anomalous samples are often available; however, usually in amounts far lower than required for a balanced classification task, and the separability assumption might not always hold. This leads to an important task—incorporating known anomalous samples into training procedures of anomaly detection models. In this work, we propose a novel model-agnostic training procedure to address this task. We reformulate one-class classification as a binary classification problem with normal data being distinguished from pseudo-anomalous samples. The pseudo-anomalous samples are drawn from low-density regions of a normalizing flow model by feeding tails of the latent distribution into the model. Such an approach allows to easily include known anomalies into the training process of an arbitrary classifier. We demonstrate that our approach shows comparable performance on one-class problems, and, most importantly, achieves comparable or superior results on tasks with variable amounts of known anomalies. PeerJ Inc. 2021-11-18 /pmc/articles/PMC8627226/ /pubmed/34901422 http://dx.doi.org/10.7717/peerj-cs.757 Text en ©2021 Ryzhikov et al. https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited. |
spellingShingle | Artificial Intelligence Ryzhikov, Artem Borisyak, Maxim Ustyuzhanin, Andrey Derkach, Denis NFAD: fixing anomaly detection using normalizing flows |
title | NFAD: fixing anomaly detection using normalizing flows |
title_full | NFAD: fixing anomaly detection using normalizing flows |
title_fullStr | NFAD: fixing anomaly detection using normalizing flows |
title_full_unstemmed | NFAD: fixing anomaly detection using normalizing flows |
title_short | NFAD: fixing anomaly detection using normalizing flows |
title_sort | nfad: fixing anomaly detection using normalizing flows |
topic | Artificial Intelligence |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8627226/ https://www.ncbi.nlm.nih.gov/pubmed/34901422 http://dx.doi.org/10.7717/peerj-cs.757 |
work_keys_str_mv | AT ryzhikovartem nfadfixinganomalydetectionusingnormalizingflows AT borisyakmaxim nfadfixinganomalydetectionusingnormalizingflows AT ustyuzhaninandrey nfadfixinganomalydetectionusingnormalizingflows AT derkachdenis nfadfixinganomalydetectionusingnormalizingflows |