Bulletin of TUIT Management and Communication Technologies, 2021
Linguistic features tokenization of text corpora of the Uzbek language
Аннотация · Abstract
The article deals with the topic of tokenization of text corpora of the Uzbek language, which should take into account the linguistic features of the spelling of the language. The types of words are analyzed and a mathematical model of word formation for complex, paired, repeated, and compound words is proposed. And also based on the models developed a finite state machine to demonstrate the spelling rules of the language, and the rules are implemented by regular expressions
Сведения · Details
- DOI 10.51348/tuitmct431
- Журнал · Journal Bulletin of TUIT Management and Communication Technologies
- Язык · Language английский
- Цитирования (Crossref) · Citations 0