Cargando…

Detection of eye contact with deep neural networks is as accurate as human experts

Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contac...

Descripción completa

Detalles Bibliográficos
Autores principales: Chong, Eunji, Clark-Whitney, Elysha, Southerland, Audrey, Stubbs, Elizabeth, Miller, Chanel, Ajodan, Eliana L., Silverman, Melanie R., Lord, Catherine, Rozga, Agata, Jones, Rebecca M., Rehg, James M.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7736573/
https://www.ncbi.nlm.nih.gov/pubmed/33318484
http://dx.doi.org/10.1038/s41467-020-19712-x
_version_ 1783622792324841472
author Chong, Eunji
Clark-Whitney, Elysha
Southerland, Audrey
Stubbs, Elizabeth
Miller, Chanel
Ajodan, Eliana L.
Silverman, Melanie R.
Lord, Catherine
Rozga, Agata
Jones, Rebecca M.
Rehg, James M.
author_facet Chong, Eunji
Clark-Whitney, Elysha
Southerland, Audrey
Stubbs, Elizabeth
Miller, Chanel
Ajodan, Eliana L.
Silverman, Melanie R.
Lord, Catherine
Rozga, Agata
Jones, Rebecca M.
Rehg, James M.
author_sort Chong, Eunji
collection PubMed
description Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers.
format Online
Article
Text
id pubmed-7736573
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-77365732020-12-28 Detection of eye contact with deep neural networks is as accurate as human experts Chong, Eunji Clark-Whitney, Elysha Southerland, Audrey Stubbs, Elizabeth Miller, Chanel Ajodan, Eliana L. Silverman, Melanie R. Lord, Catherine Rozga, Agata Jones, Rebecca M. Rehg, James M. Nat Commun Article Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers. Nature Publishing Group UK 2020-12-14 /pmc/articles/PMC7736573/ /pubmed/33318484 http://dx.doi.org/10.1038/s41467-020-19712-x Text en © The Author(s) 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
spellingShingle Article
Chong, Eunji
Clark-Whitney, Elysha
Southerland, Audrey
Stubbs, Elizabeth
Miller, Chanel
Ajodan, Eliana L.
Silverman, Melanie R.
Lord, Catherine
Rozga, Agata
Jones, Rebecca M.
Rehg, James M.
Detection of eye contact with deep neural networks is as accurate as human experts
title Detection of eye contact with deep neural networks is as accurate as human experts
title_full Detection of eye contact with deep neural networks is as accurate as human experts
title_fullStr Detection of eye contact with deep neural networks is as accurate as human experts
title_full_unstemmed Detection of eye contact with deep neural networks is as accurate as human experts
title_short Detection of eye contact with deep neural networks is as accurate as human experts
title_sort detection of eye contact with deep neural networks is as accurate as human experts
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7736573/
https://www.ncbi.nlm.nih.gov/pubmed/33318484
http://dx.doi.org/10.1038/s41467-020-19712-x
work_keys_str_mv AT chongeunji detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT clarkwhitneyelysha detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT southerlandaudrey detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT stubbselizabeth detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT millerchanel detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT ajodanelianal detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT silvermanmelanier detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT lordcatherine detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT rozgaagata detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT jonesrebeccam detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts
AT rehgjamesm detectionofeyecontactwithdeepneuralnetworksisasaccurateashumanexperts