Cargando…

HoloKinect: Holographic 3D Video Conferencing

Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D...

Descripción completa

Detalles Bibliográficos
Autores principales: Siemonsma, Stephen, Bell, Tyler
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9659293/
https://www.ncbi.nlm.nih.gov/pubmed/36365816
http://dx.doi.org/10.3390/s22218118
_version_ 1784830165591785472
author Siemonsma, Stephen
Bell, Tyler
author_facet Siemonsma, Stephen
Bell, Tyler
author_sort Siemonsma, Stephen
collection PubMed
description Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D telecommunication has not yet seen any appreciable adoption. Researchers have made great progress in developing advanced 3D telepresence systems, but often with burdensome hardware and network requirements. In this work, we present HoloKinect, an open-source, user-friendly, and GPU-accelerated platform for enabling live, two-way 3D video conferencing on commodity hardware and a standard broadband internet connection. A Microsoft Azure Kinect serves as the capture device and a Looking Glass Portrait multiscopically displays the final reconstructed 3D mesh for a hologram-like effect. HoloKinect packs color and depth information into a single video stream, leveraging multiwavelength depth (MWD) encoding to store depth maps in standard RGB video frames. The video stream is compressed with highly optimized and hardware-accelerated video codecs such as H.264. A search of the depth and video encoding parameter space was performed to analyze the quantitative and qualitative losses resulting from HoloKinect’s lossy compression scheme. Visual results were acceptable at all tested bitrates (3–30 Mbps), while the best results were achieved with higher video bitrates and full 4:4:4 chroma sampling. RMSE values of the recovered depth measurements were low across all settings permutations.
format Online
Article
Text
id pubmed-9659293
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-96592932022-11-15 HoloKinect: Holographic 3D Video Conferencing Siemonsma, Stephen Bell, Tyler Sensors (Basel) Article Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D telecommunication has not yet seen any appreciable adoption. Researchers have made great progress in developing advanced 3D telepresence systems, but often with burdensome hardware and network requirements. In this work, we present HoloKinect, an open-source, user-friendly, and GPU-accelerated platform for enabling live, two-way 3D video conferencing on commodity hardware and a standard broadband internet connection. A Microsoft Azure Kinect serves as the capture device and a Looking Glass Portrait multiscopically displays the final reconstructed 3D mesh for a hologram-like effect. HoloKinect packs color and depth information into a single video stream, leveraging multiwavelength depth (MWD) encoding to store depth maps in standard RGB video frames. The video stream is compressed with highly optimized and hardware-accelerated video codecs such as H.264. A search of the depth and video encoding parameter space was performed to analyze the quantitative and qualitative losses resulting from HoloKinect’s lossy compression scheme. Visual results were acceptable at all tested bitrates (3–30 Mbps), while the best results were achieved with higher video bitrates and full 4:4:4 chroma sampling. RMSE values of the recovered depth measurements were low across all settings permutations. MDPI 2022-10-23 /pmc/articles/PMC9659293/ /pubmed/36365816 http://dx.doi.org/10.3390/s22218118 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Siemonsma, Stephen
Bell, Tyler
HoloKinect: Holographic 3D Video Conferencing
title HoloKinect: Holographic 3D Video Conferencing
title_full HoloKinect: Holographic 3D Video Conferencing
title_fullStr HoloKinect: Holographic 3D Video Conferencing
title_full_unstemmed HoloKinect: Holographic 3D Video Conferencing
title_short HoloKinect: Holographic 3D Video Conferencing
title_sort holokinect: holographic 3d video conferencing
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9659293/
https://www.ncbi.nlm.nih.gov/pubmed/36365816
http://dx.doi.org/10.3390/s22218118
work_keys_str_mv AT siemonsmastephen holokinectholographic3dvideoconferencing
AT belltyler holokinectholographic3dvideoconferencing