关键词: 中文信息处理, 语料库, 藏文分词

Abstract:

As the fundamental linguistic knowledge base, humanannotated corpora are the basis of many statistical natural language processing tasks. Along with the wide use of statistical methods in natural language processing, corpus construction becomes an important research area.Word segmentation is necessary prerequisite of syntax parsing; its performance determines the parsing accuracy in a large degree.By the statistical analysis on a Tibetan corpus with 850,000 bytes, we first investigate the  distribution and the syntactic function of Tibetan words,  introduce a dictionarybased Tibetan word segmentation model, and then present the dictionary structure, caseauxiliary blocking and restoring algorithms which are necessary to Tibetan word segmentation. The development of the Tibetan word segmentation system also facilitates the research of the Tibetan word input methods, the Tibetan electronic dictionary construction, the Tibetan word frequency statistics, the design and realization of the search engine, the development of the machine translation system, the security of the network information, the construction of the Tibetan corpus, and the Tibetan semantic analysis.

Key words: Chinese information processing;corpus;Tibetan word segmentation