• 中国计算机学会会刊
  • 中国科技核心期刊
  • 中文核心期刊

计算机工程与科学

• 论文 • 上一篇    下一篇

基于核心词项平均划分相似度的短文本聚类算法

马慧芳,朱志强,成玉丹,贾俊杰   

  1. (西北师范大学计算机科学与工程学院,甘肃 兰州 730070)
  • 收稿日期:2016-03-24 修回日期:2016-05-13 出版日期:2017-08-25 发布日期:2017-08-25
  • 基金资助:

    国家自然科学基金(61363058);甘肃省青年科技基金(145RJYA259);甘肃省自然科学研究基金(145RJZA232,150RJZA127);中国科学院计算技术研究所智能信息处理重点实验室开放基金(IIP2014-4);2016本科生创新能力提升计划——学术科技创新团队资助项目;2016年甘肃省大学生创新创业训练计划(
    201610736040,
    201610736041)

Core term based mean partition
similarity for short text clustering

MA Hui-fang,ZHU Zhi-qiang,CHENG Yu-dan,JIA Jun-jie
 
  

  1. (College of Computer Science and Engineering,Northwest Normal University,Lanzhou 730070,China)
  • Received:2016-03-24 Revised:2016-05-13 Online:2017-08-25 Published:2017-08-25

摘要:

针对短文本特征极度稀疏、上下文依赖性强等特点,以自顶向下的策略,提出一种基于核心词项平均划分相似度的短文本聚类算法CTMPS。该方法首先在整个短文本语料库中计算词项之间的概率相关性,以此为基础对短文本中词项进行加权,将权值较大的词项作为最能代表该短文本的核心词项形成核心词项集;以信息论为基础,将核心词项作为划分依据计算平均划分相似度,选择平均划分相似度值最大包含该核心词项的短文本形成一类,用此策略反复迭代直到满足要求。最后,实验结果表明,本文提出的方法显著地提高了短文本聚类的性能。
 

关键词: 短文本聚类, 核心词项, 平均划分相似度, 概率相关性,

Abstract:

Aiming at the characteristics of extreme sparse and context dependent features of short texts, we propose a novel core term based mean partition similarity for short text clustering algorithm (CTMPS) with top-down strategy. The CTMPS firstly determines probabilistic correlation among terms in the corpus. Secondly, based on the probabilistic correlation,terms in a short text are weighted. The terms with larger weight are considered as the most representative terms of the short text and they then form the core terms set. On the basis of information theory, mean partition similarity (MPS) is calculated via core terms, and the MPS with the maximum core terms in the short text forms one class. Finally, experimental results show that the CTMPS outperforms the baseline algorithm in term of performance and clustering efficiency.
 

Key words: short text clustering, core term, mean partition similarity, probabilistic correlation, entropy