Abstract:

The valid features of microblog short texts are sparse and difficult to extract, which reduces the accuracy of text representation, classification and clustering. We propose a novel algorithm for feature selection on microblog short texts based on statistics and semantic information. We utilize Term FrequencyInverse Document Frequency (TFIDF), POS and the length of term to construct the evaluation function, and together with the semantic relevance between term and microblog short texts, the feature selection on microblog short texts is achieved, which guarantees that the selected terms can represent the meaning of microblog short texts more accurately. The new feature selection algorithm is integrated with Naive Bayesian categorization algorithm, and the experiments on an open microblog corpus show the proposed algorithm can acquire a higher precision rate of text categorization compared with the traditional strategies, indicating that the selected terms by the proposed algorithm can represent the topic of micro-blog short text more accurately.

Key words: micro-blog short text;feature selection;statistics and semantic information;POS grouping;Naive Bayesian classification algorithm