基于多属性的海量Web数据关联存储及检索系统

J4 ›› 2014, Vol. 36 ›› Issue (03): 404-410.

基于多属性的海量Web数据关联存储及检索系统

罗芳1，李春花1，周可1，黄永峰2，廖正霜1

(1.华中科技大学计算机科学与技术学院，湖北武汉 430074;2.清华大学电子工程系，北京 100084)

收稿日期:2013-06-08 修回日期:2013-10-20 出版日期:2014-03-25 发布日期:2014-03-25
基金资助:
国家863计划资助项目(2012AA011004)；清华大学自主科研项目基金(20111081023)

An associated storage and retrieval system of massive
Web data based on multi-attributes

LUO Fang1,LI Chunhua1,ZHOU Ke1,HUANG Yongfeng2,LIAO Zhengshuang1

(1.School of Computer Science and Technology,Huazhong University of Science and Technology,Wuhan 430074;
2.Department of Electronic Engineering,Tsinghua University,Beijing 100084,China)

Received:2013-06-08 Revised:2013-10-20 Online:2014-03-25 Published:2014-03-25

摘要/Abstract

摘要：

传统的Web数据检索一般采用全文检索方法，该方法具有很好的灵活性，但舆情分析往往需要获得相关的网页属性及统计信息。针对传统的Web检索方法无法满足上述需求，基于Hadoop平台设计并实现了一种基于多属性的海量Web数据的关联存储及检索系统，为舆情分析提供基础检索与统计服务。主要实现HDFS上基于属性的网页数据的分类和聚类存储，解决小文件存储同时提高数据访问吞吐量；建立原始网页数据与属性数据之间的关联映射；基于HBase的已有索引机制，结合分布式本地索引机制解决基于HBase的动态属性多条件选择查询的辅助索引问题。

关键词: 分类存储, 多条件选择查询, 关联映射, 辅助索引

Abstract:

Traditional Web Retrievals commonly use the fulltext search method which has good flexibility. However, as the analysis of public opinion usually needs relative information of web attributes and statistics, the traditional retrieval method can not satisfy it well. An associated storage and retrieval system based on the Hadoop platform is designed and implemented, which can offer good basic service for the analysis of public opinion. Firstly, the associated storage of web data based on HDFS is realized by machine learning. Secondly, the problem of small files storage together with the access efficiency of associated data is solved. Thirdly, the mapping between original web data and the extracted attributes is established. Finally, the retrieval of dynamic multiple attributes based on the existed indexing on HBase and the distributed local indexing are realized.

Key words: category storage;multiconditions selectable query;associated mapping;secondary indexing

罗芳1，李春花1，周可1，黄永峰2，廖正霜1. 基于多属性的海量Web数据关联存储及检索系统[J]. J4, 2014, 36(03): 404-410.

LUO Fang1,LI Chunhua1,ZHOU Ke1,HUANG Yongfeng2,LIAO Zhengshuang1. An associated storage and retrieval system of massive
Web data based on multi-attributes [J]. J4, 2014, 36(03): 404-410.

参考文献

［1］Apache Hadoop project［EB/OL］.［20120910］. http://hadoop.apache.org.
［2］Apache HBase project ［EB/OL］.［20121125］. http://hb ase.apache.org.
［3］Liu Xuhui, Han Jizhong, Zhong Yunqin, et al. Implementing WebGIS on Hadoop:A case study of improving small file I/O performance on HDFS［C］∥Proc of CLUSTER, 2009:18.
［4］Dong Bo, Qiu Jie, Zheng Qinghua, et al. A novel approach to improving the efficiency of storing and accessing small files on Hadoop:A case study by PowerPoint files［C］∥Proc of the 2010 IEEE International Conference on Services Computing, 2010:6572.
［5］Qian Aibing. A model for analyzing public opinion under the web and its implementation［J］. Information Analysis and Research, 2008(4):4955.(in Chinese)
［6］Zhang Xuegong. Pattern recognition［M］. 3rd ed. Beijing:Tsinghua University Press, 2010.(in Chinese)
［7］Hbasetransactionaltableindexed［EB/OL］.［20021026］. https://github.com/hbasetrx/hbasetransactionaltableindexed.
［8］Meng Biping, Wang Tengjiao, Li Hongyan, et al. Regional bitmap index:A secondary index for data management in could computing environment［J］. Chinese Journal of Computers,2012,35(11):23062316.(in Chinese)
［9］Wang Mingyan. Keyword and key concept extraction technique based on web page［D］. Beijing:Beijing University of Technology,2003.(in Chinese)
［10］Jiang Xiaoping, Li Chenghua, Xiang Wen, et al. Parallel implementing kmeans clustering algorithm using MapReduce programming mode［J］. Journal of Huazhong University of Science and Technology(Natural Science Edition),2011,39( Sup Ⅰ):120124.(in Chinese)
［11］Yu Ge, Gu Yu, Bao Yubin, et al. Large scale graph data processing on cloud computing environments［J］. Chinese Journal of Computers,2011,34(10):17531767.(in Chinese)
［12］He Yunfeng, Yu Junqing, Tang Jiufei, et al. Video data organization and management based on MPEG7［J］. Journal of Wuhan University(Natural Science Edition),2010, 56(6):711716.(in Chinese)
附中文参考文献:
［5］钱爱兵.基于主题的网络舆情分析模型及其实现［J］. 情报分析与研究, 2008(4):4955.
［6］张学工.模式识别［M］.第3版. 北京：清华大学出版社,2010.
［8］孟必平,王腾蛟,李红燕,等. 分片位图索引:一种使用于云数据管理的辅助索引机制［J］. 计算机学报, 2012,35(11):23062316.
［9］王明燕,基于WEB页面的关键词与关键概念提取技术［D］. 北京:北京工业大学, 2003.
［10］江小平,李成华,向文,等. kmeans聚类算法的MapReduce并行化实现［J］. 华中科技大学学报（自然科学版）,2011，（Sup Ⅰ）39:120124.
［11］于戈,谷峪,鲍玉斌,等. 云计算环境下的大规模图数据处理技术［J］.计算机学报,2011,34(10):17531767.
［12］何云峰,于俊清,唐九飞,等. 基于MPEG7的视频数据组织与管理［J］. 武汉大学学报(理学版) ,2010,56(6):711716.