Cargando…

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test–retest reliability, sug...

Descripción completa

Detalles Bibliográficos
Autores principales: Yu, Sigang, Shi, Enze, Wang, Ruoyang, Zhao, Shijie, Liu, Tianming, Jiang, Xi, Zhang, Shu
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9563232/
https://www.ncbi.nlm.nih.gov/pubmed/36248685
http://dx.doi.org/10.3389/fnhum.2022.944543
_version_ 1784808352174309376
author Yu, Sigang
Shi, Enze
Wang, Ruoyang
Zhao, Shijie
Liu, Tianming
Jiang, Xi
Zhang, Shu
author_facet Yu, Sigang
Shi, Enze
Wang, Ruoyang
Zhao, Shijie
Liu, Tianming
Jiang, Xi
Zhang, Shu
author_sort Yu, Sigang
collection PubMed
description Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test–retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the “semantic gap” between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.
format Online
Article
Text
id pubmed-9563232
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-95632322022-10-15 A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging Yu, Sigang Shi, Enze Wang, Ruoyang Zhao, Shijie Liu, Tianming Jiang, Xi Zhang, Shu Front Hum Neurosci Neuroscience Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test–retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the “semantic gap” between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application. Frontiers Media S.A. 2022-09-30 /pmc/articles/PMC9563232/ /pubmed/36248685 http://dx.doi.org/10.3389/fnhum.2022.944543 Text en Copyright © 2022 Yu, Shi, Wang, Zhao, Liu, Jiang and Zhang. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Neuroscience
Yu, Sigang
Shi, Enze
Wang, Ruoyang
Zhao, Shijie
Liu, Tianming
Jiang, Xi
Zhang, Shu
A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title_full A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title_fullStr A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title_full_unstemmed A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title_short A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
title_sort hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9563232/
https://www.ncbi.nlm.nih.gov/pubmed/36248685
http://dx.doi.org/10.3389/fnhum.2022.944543
work_keys_str_mv AT yusigang ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT shienze ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT wangruoyang ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT zhaoshijie ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT liutianming ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT jiangxi ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT zhangshu ahybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT yusigang hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT shienze hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT wangruoyang hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT zhaoshijie hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT liutianming hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT jiangxi hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging
AT zhangshu hybridlearningframeworkforfinegrainedinterpretationofbrainspatiotemporalpatternsduringnaturalisticfunctionalmagneticresonanceimaging