Morpheme Matching Based Text Tokenization for a Scarce Resourced Language

Text tokenization is a fundamental pre-processing step for almost all the information processing applications. This task is nontrivial for the scarce resourced languages such as Urdu, as there is inconsistent use of space between words. In this paper a morpheme matching based approach has been propo...

Descripción completa

Detalles Bibliográficos
Autores principales: Rehman, Zobia, Anwar, Waqas, Bajwa, Usama Ijaz, Xuan, Wang, Chaoying, Zhou
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2013
Materias:
Research Article
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3749178/
https://www.ncbi.nlm.nih.gov/pubmed/23990871
http://dx.doi.org/10.1371/journal.pone.0068178
Descripción
Sumario:Text tokenization is a fundamental pre-processing step for almost all the information processing applications. This task is nontrivial for the scarce resourced languages such as Urdu, as there is inconsistent use of space between words. In this paper a morpheme matching based approach has been proposed for Urdu text tokenization, along with some other algorithms to solve the additional issues of boundary detection of compound words, affixation, reduplication, names and abbreviations. This study resulted into 97.28% precision, 93.71% recall, and 95.46% F1-measure; while tokenizing a corpus of 57000 words by using a morpheme list with 6400 entries.

Ejemplares similares