Cargando…

NFAD: fixing anomaly detection using normalizing flows

Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-cla...

Descripción completa

Detalles Bibliográficos
Autores principales: Ryzhikov, Artem, Borisyak, Maxim, Ustyuzhanin, Andrey, Derkach, Denis
Formato: Online Artículo Texto
Lenguaje:English
Publicado: PeerJ Inc. 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8627226/
https://www.ncbi.nlm.nih.gov/pubmed/34901422
http://dx.doi.org/10.7717/peerj-cs.757
_version_ 1784606810882179072
author Ryzhikov, Artem
Borisyak, Maxim
Ustyuzhanin, Andrey
Derkach, Denis
author_facet Ryzhikov, Artem
Borisyak, Maxim
Ustyuzhanin, Andrey
Derkach, Denis
author_sort Ryzhikov, Artem
collection PubMed
description Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-class SVM and Robust Auto-Encoder, are one-class classification methods, i.e., focus on separating normal data from the rest of the space. Such methods are based on the assumption of separability of normal and anomalous classes, and subsequently do not take into account any available samples of anomalies. Nonetheless, in practical settings, some anomalous samples are often available; however, usually in amounts far lower than required for a balanced classification task, and the separability assumption might not always hold. This leads to an important task—incorporating known anomalous samples into training procedures of anomaly detection models. In this work, we propose a novel model-agnostic training procedure to address this task. We reformulate one-class classification as a binary classification problem with normal data being distinguished from pseudo-anomalous samples. The pseudo-anomalous samples are drawn from low-density regions of a normalizing flow model by feeding tails of the latent distribution into the model. Such an approach allows to easily include known anomalies into the training process of an arbitrary classifier. We demonstrate that our approach shows comparable performance on one-class problems, and, most importantly, achieves comparable or superior results on tasks with variable amounts of known anomalies.
format Online
Article
Text
id pubmed-8627226
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher PeerJ Inc.
record_format MEDLINE/PubMed
spelling pubmed-86272262021-12-10 NFAD: fixing anomaly detection using normalizing flows Ryzhikov, Artem Borisyak, Maxim Ustyuzhanin, Andrey Derkach, Denis PeerJ Comput Sci Artificial Intelligence Anomaly detection is a challenging task that frequently arises in practically all areas of industry and science, from fraud detection and data quality monitoring to finding rare cases of diseases and searching for new physics. Most of the conventional approaches to anomaly detection, such as one-class SVM and Robust Auto-Encoder, are one-class classification methods, i.e., focus on separating normal data from the rest of the space. Such methods are based on the assumption of separability of normal and anomalous classes, and subsequently do not take into account any available samples of anomalies. Nonetheless, in practical settings, some anomalous samples are often available; however, usually in amounts far lower than required for a balanced classification task, and the separability assumption might not always hold. This leads to an important task—incorporating known anomalous samples into training procedures of anomaly detection models. In this work, we propose a novel model-agnostic training procedure to address this task. We reformulate one-class classification as a binary classification problem with normal data being distinguished from pseudo-anomalous samples. The pseudo-anomalous samples are drawn from low-density regions of a normalizing flow model by feeding tails of the latent distribution into the model. Such an approach allows to easily include known anomalies into the training process of an arbitrary classifier. We demonstrate that our approach shows comparable performance on one-class problems, and, most importantly, achieves comparable or superior results on tasks with variable amounts of known anomalies. PeerJ Inc. 2021-11-18 /pmc/articles/PMC8627226/ /pubmed/34901422 http://dx.doi.org/10.7717/peerj-cs.757 Text en ©2021 Ryzhikov et al. https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited.
spellingShingle Artificial Intelligence
Ryzhikov, Artem
Borisyak, Maxim
Ustyuzhanin, Andrey
Derkach, Denis
NFAD: fixing anomaly detection using normalizing flows
title NFAD: fixing anomaly detection using normalizing flows
title_full NFAD: fixing anomaly detection using normalizing flows
title_fullStr NFAD: fixing anomaly detection using normalizing flows
title_full_unstemmed NFAD: fixing anomaly detection using normalizing flows
title_short NFAD: fixing anomaly detection using normalizing flows
title_sort nfad: fixing anomaly detection using normalizing flows
topic Artificial Intelligence
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8627226/
https://www.ncbi.nlm.nih.gov/pubmed/34901422
http://dx.doi.org/10.7717/peerj-cs.757
work_keys_str_mv AT ryzhikovartem nfadfixinganomalydetectionusingnormalizingflows
AT borisyakmaxim nfadfixinganomalydetectionusingnormalizingflows
AT ustyuzhaninandrey nfadfixinganomalydetectionusingnormalizingflows
AT derkachdenis nfadfixinganomalydetectionusingnormalizingflows