Cargando…

Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging

BACKGROUND: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera–based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monit...

Descripción completa

Detalles Bibliográficos
Autores principales: Cho, Youngjun, Julier, Simon J, Bianchi-Berthouze, Nadia
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6477570/
https://www.ncbi.nlm.nih.gov/pubmed/30964440
http://dx.doi.org/10.2196/10140
_version_ 1783413041400905728
author Cho, Youngjun
Julier, Simon J
Bianchi-Berthouze, Nadia
author_facet Cho, Youngjun
Julier, Simon J
Bianchi-Berthouze, Nadia
author_sort Cho, Youngjun
collection PubMed
description BACKGROUND: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera–based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monitoring results, a person must remain still for several minutes while a measurement is being taken. This is cumbersome and makes its use in real-life situations impractical. OBJECTIVE: We proposed a system that combines PPG and thermography with the aim of improving cardiovascular signal quality and detecting stress responses quickly. METHODS: Using a smartphone camera with a low-cost thermal camera added on, we built a novel system that continuously and reliably measures 2 different types of cardiovascular events: (1) blood volume pulse and (2) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 participants, involved in stress-inducing mental workload tasks, measured their physiological responses to stressors over a short time period (20 seconds) immediately after each task. Participants reported their perceived stress levels on a 10-cm visual analog scale. For the instant stress inference task, we built novel low-level feature sets representing cardiovascular variability. We then used the automatic feature learning capability of artificial neural networks to improve the mapping between the extracted features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. RESULTS: First, we found that the measured PPG signals presented high quality cardiac cyclic information (mean pSQI: 0.755; SD 0.068). We also found that the measured thermal changes of the nose tip presented high-quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (mean pSQI: from 0.714 to 0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the cardiovascular signals (ie, heart rate variability and thermal directionality) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing precrafted features-based methods. In addition, the 17-fold leave-one-subject-out cross-validation results showed that combining both modalities produced higher accuracy than using PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art stress recognition methods that require long-term measurements. Finally, we explored effects of different data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. CONCLUSIONS: The results demonstrate the feasibility of using smartphone-based imaging for instant stress detection. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental health care solutions in the wild.
format Online
Article
Text
id pubmed-6477570
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-64775702019-05-08 Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging Cho, Youngjun Julier, Simon J Bianchi-Berthouze, Nadia JMIR Ment Health Original Paper BACKGROUND: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera–based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monitoring results, a person must remain still for several minutes while a measurement is being taken. This is cumbersome and makes its use in real-life situations impractical. OBJECTIVE: We proposed a system that combines PPG and thermography with the aim of improving cardiovascular signal quality and detecting stress responses quickly. METHODS: Using a smartphone camera with a low-cost thermal camera added on, we built a novel system that continuously and reliably measures 2 different types of cardiovascular events: (1) blood volume pulse and (2) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 participants, involved in stress-inducing mental workload tasks, measured their physiological responses to stressors over a short time period (20 seconds) immediately after each task. Participants reported their perceived stress levels on a 10-cm visual analog scale. For the instant stress inference task, we built novel low-level feature sets representing cardiovascular variability. We then used the automatic feature learning capability of artificial neural networks to improve the mapping between the extracted features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. RESULTS: First, we found that the measured PPG signals presented high quality cardiac cyclic information (mean pSQI: 0.755; SD 0.068). We also found that the measured thermal changes of the nose tip presented high-quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (mean pSQI: from 0.714 to 0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the cardiovascular signals (ie, heart rate variability and thermal directionality) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing precrafted features-based methods. In addition, the 17-fold leave-one-subject-out cross-validation results showed that combining both modalities produced higher accuracy than using PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art stress recognition methods that require long-term measurements. Finally, we explored effects of different data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. CONCLUSIONS: The results demonstrate the feasibility of using smartphone-based imaging for instant stress detection. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental health care solutions in the wild. JMIR Publications 2019-04-09 /pmc/articles/PMC6477570/ /pubmed/30964440 http://dx.doi.org/10.2196/10140 Text en ©Youngjun Cho, Simon J Julier, Nadia Bianchi-Berthouze. Originally published in JMIR Mental Health (http://mental.jmir.org), 09.04.2019. https://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
spellingShingle Original Paper
Cho, Youngjun
Julier, Simon J
Bianchi-Berthouze, Nadia
Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title_full Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title_fullStr Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title_full_unstemmed Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title_short Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging
title_sort instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging
topic Original Paper
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6477570/
https://www.ncbi.nlm.nih.gov/pubmed/30964440
http://dx.doi.org/10.2196/10140
work_keys_str_mv AT choyoungjun instantstressdetectionofperceivedmentalstressthroughsmartphonephotoplethysmographyandthermalimaging
AT juliersimonj instantstressdetectionofperceivedmentalstressthroughsmartphonephotoplethysmographyandthermalimaging
AT bianchiberthouzenadia instantstressdetectionofperceivedmentalstressthroughsmartphonephotoplethysmographyandthermalimaging