关键词: 统计分词, 词典, 特征提取

Abstract: This paper analyzes the various segmentation methods in the information processing technology.In view of the current segmentation methods in the network which do not recognize the new emerging words,we design a new subword method based on statistics. This method avoids complex grammar and rules, needs no enormous support from dictionaries, and resolves the problems brought by the new words. So we conclude that this method has better exactness and is very pragmatic and powerful in practical operations.

Key words: web;statistical word segmentation;dictionary;feature selection