Cargando…

Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations

Nonword pronunciation is a critical challenge for models of reading aloud but little attention has been given to identifying the best method for assessing model predictions. The most typical approach involves comparing the model’s pronunciations of nonwords to pronunciations of the same nonwords by...

Descripción completa

Detalles Bibliográficos
Autores principales: Gubian, Michele, Blything, Ryan, Davis, Colin J., Bowers, Jeffrey S.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer US 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10126079/
https://www.ncbi.nlm.nih.gov/pubmed/35650383
http://dx.doi.org/10.3758/s13428-022-01794-8
_version_ 1785030160256335872
author Gubian, Michele
Blything, Ryan
Davis, Colin J.
Bowers, Jeffrey S.
author_facet Gubian, Michele
Blything, Ryan
Davis, Colin J.
Bowers, Jeffrey S.
author_sort Gubian, Michele
collection PubMed
description Nonword pronunciation is a critical challenge for models of reading aloud but little attention has been given to identifying the best method for assessing model predictions. The most typical approach involves comparing the model’s pronunciations of nonwords to pronunciations of the same nonwords by human participants and deeming the model’s output correct if it matches with any transcription of the human pronunciations. The present paper introduces a new ratings-based method, in which participants are shown printed nonwords and asked to rate the plausibility of the provided pronunciations, generated here by a speech synthesiser. We demonstrate this method with reference to a previously published database of 915 disyllabic nonwords (Mousikou et al., 2017). We evaluated two well-known psychological models, RC00 and CDP++, as well as an additional grapheme-to-phoneme algorithm known as Sequitur, and compared our model assessment with the corpus-based method adopted by Mousikou et al. We find that the ratings method: a) is much easier to implement than a corpus-based method, b) has a high hit rate and low false-alarm rate in assessing nonword reading accuracy, and c) provided a similar outcome as the corpus-based method in its assessment of RC00 and CDP++. However, the two methods differed in their evaluation of Sequitur, which performed much better under the ratings method. Indeed, our evaluation of Sequitur revealed that the corpus-based method introduced a number of false positives and more often, false negatives. Implications of these findings are discussed. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.3758/s13428-022-01794-8.
format Online
Article
Text
id pubmed-10126079
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Springer US
record_format MEDLINE/PubMed
spelling pubmed-101260792023-04-26 Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations Gubian, Michele Blything, Ryan Davis, Colin J. Bowers, Jeffrey S. Behav Res Methods Article Nonword pronunciation is a critical challenge for models of reading aloud but little attention has been given to identifying the best method for assessing model predictions. The most typical approach involves comparing the model’s pronunciations of nonwords to pronunciations of the same nonwords by human participants and deeming the model’s output correct if it matches with any transcription of the human pronunciations. The present paper introduces a new ratings-based method, in which participants are shown printed nonwords and asked to rate the plausibility of the provided pronunciations, generated here by a speech synthesiser. We demonstrate this method with reference to a previously published database of 915 disyllabic nonwords (Mousikou et al., 2017). We evaluated two well-known psychological models, RC00 and CDP++, as well as an additional grapheme-to-phoneme algorithm known as Sequitur, and compared our model assessment with the corpus-based method adopted by Mousikou et al. We find that the ratings method: a) is much easier to implement than a corpus-based method, b) has a high hit rate and low false-alarm rate in assessing nonword reading accuracy, and c) provided a similar outcome as the corpus-based method in its assessment of RC00 and CDP++. However, the two methods differed in their evaluation of Sequitur, which performed much better under the ratings method. Indeed, our evaluation of Sequitur revealed that the corpus-based method introduced a number of false positives and more often, false negatives. Implications of these findings are discussed. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.3758/s13428-022-01794-8. Springer US 2022-06-01 2023 /pmc/articles/PMC10126079/ /pubmed/35650383 http://dx.doi.org/10.3758/s13428-022-01794-8 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Gubian, Michele
Blything, Ryan
Davis, Colin J.
Bowers, Jeffrey S.
Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title_full Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title_fullStr Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title_full_unstemmed Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title_short Does that sound right? A novel method of evaluating models of reading aloud: Rating nonword pronunciations
title_sort does that sound right? a novel method of evaluating models of reading aloud: rating nonword pronunciations
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10126079/
https://www.ncbi.nlm.nih.gov/pubmed/35650383
http://dx.doi.org/10.3758/s13428-022-01794-8
work_keys_str_mv AT gubianmichele doesthatsoundrightanovelmethodofevaluatingmodelsofreadingaloudratingnonwordpronunciations
AT blythingryan doesthatsoundrightanovelmethodofevaluatingmodelsofreadingaloudratingnonwordpronunciations
AT daviscolinj doesthatsoundrightanovelmethodofevaluatingmodelsofreadingaloudratingnonwordpronunciations
AT bowersjeffreys doesthatsoundrightanovelmethodofevaluatingmodelsofreadingaloudratingnonwordpronunciations