Cargando…

Multimodal Federated Learning: A Survey

Federated learning (FL), which provides a collaborative training scheme for distributed data sources with privacy concerns, has become a burgeoning and attractive research area. Most existing FL studies focus on taking unimodal data, such as image and text, as the model input and resolving the heter...

Descripción completa

Detalles Bibliográficos
Autores principales: Che, Liwei, Wang, Jiaqi, Zhou, Yao, Ma, Fenglong
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10422520/
https://www.ncbi.nlm.nih.gov/pubmed/37571768
http://dx.doi.org/10.3390/s23156986
_version_ 1785089230789148672
author Che, Liwei
Wang, Jiaqi
Zhou, Yao
Ma, Fenglong
author_facet Che, Liwei
Wang, Jiaqi
Zhou, Yao
Ma, Fenglong
author_sort Che, Liwei
collection PubMed
description Federated learning (FL), which provides a collaborative training scheme for distributed data sources with privacy concerns, has become a burgeoning and attractive research area. Most existing FL studies focus on taking unimodal data, such as image and text, as the model input and resolving the heterogeneity challenge, i.e., the challenge of non-identical distribution (non-IID) caused by a data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been conducted to improve system performance utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic of multimodal federated learning (MFL) and present a literature review on the state-of-art MFL methods. Furthermore, we categorize multimodal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research.
format Online
Article
Text
id pubmed-10422520
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-104225202023-08-13 Multimodal Federated Learning: A Survey Che, Liwei Wang, Jiaqi Zhou, Yao Ma, Fenglong Sensors (Basel) Review Federated learning (FL), which provides a collaborative training scheme for distributed data sources with privacy concerns, has become a burgeoning and attractive research area. Most existing FL studies focus on taking unimodal data, such as image and text, as the model input and resolving the heterogeneity challenge, i.e., the challenge of non-identical distribution (non-IID) caused by a data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been conducted to improve system performance utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic of multimodal federated learning (MFL) and present a literature review on the state-of-art MFL methods. Furthermore, we categorize multimodal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research. MDPI 2023-08-06 /pmc/articles/PMC10422520/ /pubmed/37571768 http://dx.doi.org/10.3390/s23156986 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Review
Che, Liwei
Wang, Jiaqi
Zhou, Yao
Ma, Fenglong
Multimodal Federated Learning: A Survey
title Multimodal Federated Learning: A Survey
title_full Multimodal Federated Learning: A Survey
title_fullStr Multimodal Federated Learning: A Survey
title_full_unstemmed Multimodal Federated Learning: A Survey
title_short Multimodal Federated Learning: A Survey
title_sort multimodal federated learning: a survey
topic Review
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10422520/
https://www.ncbi.nlm.nih.gov/pubmed/37571768
http://dx.doi.org/10.3390/s23156986
work_keys_str_mv AT cheliwei multimodalfederatedlearningasurvey
AT wangjiaqi multimodalfederatedlearningasurvey
AT zhouyao multimodalfederatedlearningasurvey
AT mafenglong multimodalfederatedlearningasurvey