Cargando…
Detection of eye contact with deep neural networks is as accurate as human experts
Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contac...
Autores principales: | , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Nature Publishing Group UK
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7736573/ https://www.ncbi.nlm.nih.gov/pubmed/33318484 http://dx.doi.org/10.1038/s41467-020-19712-x |
_version_ | 1783622792324841472 |
---|---|
author | Chong, Eunji Clark-Whitney, Elysha Southerland, Audrey Stubbs, Elizabeth Miller, Chanel Ajodan, Eliana L. Silverman, Melanie R. Lord, Catherine Rozga, Agata Jones, Rebecca M. Rehg, James M. |
author_facet | Chong, Eunji Clark-Whitney, Elysha Southerland, Audrey Stubbs, Elizabeth Miller, Chanel Ajodan, Eliana L. Silverman, Melanie R. Lord, Catherine Rozga, Agata Jones, Rebecca M. Rehg, James M. |
author_sort | Chong, Eunji |
collection | PubMed |
description | Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers. |
format | Online Article Text |
id | pubmed-7736573 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Nature Publishing Group UK |
record_format | MEDLINE/PubMed |
spelling | pubmed-77365732020-12-28 Detection of eye contact with deep neural networks is as accurate as human experts Chong, Eunji Clark-Whitney, Elysha Southerland, Audrey Stubbs, Elizabeth Miller, Chanel Ajodan, Eliana L. Silverman, Melanie R. Lord, Catherine Rozga, Agata Jones, Rebecca M. Rehg, James M. Nat Commun Article Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers. Nature Publishing Group UK 2020-12-14 /pmc/articles/PMC7736573/ /pubmed/33318484 http://dx.doi.org/10.1038/s41467-020-19712-x Text en © The Author(s) 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. |
spellingShingle | Article Chong, Eunji Clark-Whitney, Elysha Southerland, Audrey Stubbs, Elizabeth Miller, Chanel Ajodan, Eliana L. Silverman, Melanie R. Lord, Catherine Rozga, Agata Jones, Rebecca M. Rehg, James M. Detection of eye contact with deep neural networks is as accurate as human experts |
title | Detection of eye contact with deep neural networks is as accurate as human experts |
title_full | Detection of eye contact with deep neural networks is as accurate as human experts |
title_fullStr | Detection of eye contact with deep neural networks is as accurate as human experts |
title_full_unstemmed | Detection of eye contact with deep neural networks is as accurate as human experts |
title_short | Detection of eye contact with deep neural networks is as accurate as human experts |
title_sort | detection of eye contact with deep neural networks is as accurate as human experts |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7736573/ https://www.ncbi.nlm.nih.gov/pubmed/33318484 http://dx.doi.org/10.1038/s41467-020-19712-x |
work_keys_str_mv | AT chongeunji detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT clarkwhitneyelysha detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT southerlandaudrey detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT stubbselizabeth detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT millerchanel detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT ajodanelianal detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT silvermanmelanier detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT lordcatherine detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT rozgaagata detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT jonesrebeccam detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts AT rehgjamesm detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts |