Cargando…

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones

Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing te...

Descripción completa

Detalles Bibliográficos
Autores principales: Wang, Shuang, Liu, Jingyu, Lan, Xuedan, Hu, Qihang, Jiang, Jian, Zhang, Jingjing
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9763593/
https://www.ncbi.nlm.nih.gov/pubmed/36562072
http://dx.doi.org/10.3389/fpsyg.2022.970219
_version_ 1784853093787107328
author Wang, Shuang
Liu, Jingyu
Lan, Xuedan
Hu, Qihang
Jiang, Jian
Zhang, Jingjing
author_facet Wang, Shuang
Liu, Jingyu
Lan, Xuedan
Hu, Qihang
Jiang, Jian
Zhang, Jingjing
author_sort Wang, Shuang
collection PubMed
description Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely “cold/warm,” “soft/hard,” “transparent/turbid,” “far/near,” “weak/strong,” pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.
format Online
Article
Text
id pubmed-9763593
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-97635932022-12-21 Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones Wang, Shuang Liu, Jingyu Lan, Xuedan Hu, Qihang Jiang, Jian Zhang, Jingjing Front Psychol Psychology Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely “cold/warm,” “soft/hard,” “transparent/turbid,” “far/near,” “weak/strong,” pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research. Frontiers Media S.A. 2022-12-06 /pmc/articles/PMC9763593/ /pubmed/36562072 http://dx.doi.org/10.3389/fpsyg.2022.970219 Text en Copyright © 2022 Wang, Liu, Lan, Hu, Jiang and Zhang. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Wang, Shuang
Liu, Jingyu
Lan, Xuedan
Hu, Qihang
Jiang, Jian
Zhang, Jingjing
Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title_full Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title_fullStr Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title_full_unstemmed Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title_short Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
title_sort cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9763593/
https://www.ncbi.nlm.nih.gov/pubmed/36562072
http://dx.doi.org/10.3389/fpsyg.2022.970219
work_keys_str_mv AT wangshuang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones
AT liujingyu crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones
AT lanxuedan crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones
AT huqihang crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones
AT jiangjian crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones
AT zhangjingjing crossmodalassociationanalysisandmatchingmodelconstructionofperceptualattributesofmultiplecolorsandcombinedtones