Cargando…
Speech synthesis from neural decoding of spoken sentences
Technology that translates neural activity into speech would be transformative for people unable to communicate as a result of neurological impairment. Decoding speech from neural activity is challenging because speaking requires such precise and rapid multi-dimensional control of vocal tract articu...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
2019
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9714519/ https://www.ncbi.nlm.nih.gov/pubmed/31019317 http://dx.doi.org/10.1038/s41586-019-1119-1 |
_version_ | 1784842246630146048 |
---|---|
author | Anumanchipalli, Gopala K. Chartier, Josh Chang, Edward F. |
author_facet | Anumanchipalli, Gopala K. Chartier, Josh Chang, Edward F. |
author_sort | Anumanchipalli, Gopala K. |
collection | PubMed |
description | Technology that translates neural activity into speech would be transformative for people unable to communicate as a result of neurological impairment. Decoding speech from neural activity is challenging because speaking requires such precise and rapid multi-dimensional control of vocal tract articulators. Here, we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into articulatory movement representations, and then transformed those representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe neurally synthesized speech. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of speech neuroprosthetic technology to restore spoken communication. |
format | Online Article Text |
id | pubmed-9714519 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2019 |
record_format | MEDLINE/PubMed |
spelling | pubmed-97145192022-12-01 Speech synthesis from neural decoding of spoken sentences Anumanchipalli, Gopala K. Chartier, Josh Chang, Edward F. Nature Article Technology that translates neural activity into speech would be transformative for people unable to communicate as a result of neurological impairment. Decoding speech from neural activity is challenging because speaking requires such precise and rapid multi-dimensional control of vocal tract articulators. Here, we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into articulatory movement representations, and then transformed those representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe neurally synthesized speech. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of speech neuroprosthetic technology to restore spoken communication. 2019-04 2019-04-24 /pmc/articles/PMC9714519/ /pubmed/31019317 http://dx.doi.org/10.1038/s41586-019-1119-1 Text en http://www.nature.com/authors/editorial_policies/license.html#termsUsers may view, print, copy, and download text and data-mine the content in such documents, for the purposes of academic research, subject always to the full Conditions of use:http://www.nature.com/authors/editorial_policies/license.html#terms |
spellingShingle | Article Anumanchipalli, Gopala K. Chartier, Josh Chang, Edward F. Speech synthesis from neural decoding of spoken sentences |
title | Speech synthesis from neural decoding of spoken sentences |
title_full | Speech synthesis from neural decoding of spoken sentences |
title_fullStr | Speech synthesis from neural decoding of spoken sentences |
title_full_unstemmed | Speech synthesis from neural decoding of spoken sentences |
title_short | Speech synthesis from neural decoding of spoken sentences |
title_sort | speech synthesis from neural decoding of spoken sentences |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9714519/ https://www.ncbi.nlm.nih.gov/pubmed/31019317 http://dx.doi.org/10.1038/s41586-019-1119-1 |
work_keys_str_mv | AT anumanchipalligopalak speechsynthesisfromneuraldecodingofspokensentences AT chartierjosh speechsynthesisfromneuraldecodingofspokensentences AT changedwardf speechsynthesisfromneuraldecodingofspokensentences |