Cargando…

Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen

Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited,...

Descripción completa

Detalles Bibliográficos
Autores principales: Wehbi, Mohamad, Luge, Daniel, Hamann, Tim, Barth, Jens, Kaempf, Peter, Zanca, Dario, Eskofier, Bjoern M.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9318904/
https://www.ncbi.nlm.nih.gov/pubmed/35891027
http://dx.doi.org/10.3390/s22145347
_version_ 1784755418016251904
author Wehbi, Mohamad
Luge, Daniel
Hamann, Tim
Barth, Jens
Kaempf, Peter
Zanca, Dario
Eskofier, Bjoern M.
author_facet Wehbi, Mohamad
Luge, Daniel
Hamann, Tim
Barth, Jens
Kaempf, Peter
Zanca, Dario
Eskofier, Bjoern M.
author_sort Wehbi, Mohamad
collection PubMed
description Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%.
format Online
Article
Text
id pubmed-9318904
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-93189042022-07-27 Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen Wehbi, Mohamad Luge, Daniel Hamann, Tim Barth, Jens Kaempf, Peter Zanca, Dario Eskofier, Bjoern M. Sensors (Basel) Article Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%. MDPI 2022-07-18 /pmc/articles/PMC9318904/ /pubmed/35891027 http://dx.doi.org/10.3390/s22145347 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Wehbi, Mohamad
Luge, Daniel
Hamann, Tim
Barth, Jens
Kaempf, Peter
Zanca, Dario
Eskofier, Bjoern M.
Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title_full Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title_fullStr Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title_full_unstemmed Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title_short Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
title_sort surface-free multi-stroke trajectory reconstruction and word recognition using an imu-enhanced digital pen
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9318904/
https://www.ncbi.nlm.nih.gov/pubmed/35891027
http://dx.doi.org/10.3390/s22145347
work_keys_str_mv AT wehbimohamad surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT lugedaniel surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT hamanntim surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT barthjens surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT kaempfpeter surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT zancadario surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen
AT eskofierbjoernm surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen