集成I/O硬件压缩加速器的Hadoop系统结构

计算机工程与科学

集成I/O硬件压缩加速器的Hadoop系统结构

雷力1，钱斌海1，郭俊1，顾雄礼2，刘鹏1

（1.浙江大学信息与电子工程学院,浙江杭州 310027;2.华为技术有限公司,浙江杭州 310051）

收稿日期:2016-04-20 修回日期:2016-06-15 出版日期:2016-08-25 发布日期:2016-08-25
基金资助:
华为技术有限公司资助项目（YB2014100047）

Integrated I/O hardware compression accelerators of Hadoop system architecture

LEI Li1,QIAN Binhai1,GUO Jun1,GU Xiongli2,LIU Peng1

(1.College of Information Science & Electronic Engineering,Zhejiang University,Hangzhou 310027;
2.Huawei Technologies Co.,Ltd.,Hangzhou 310051,China)

Received:2016-04-20 Revised:2016-06-15 Online:2016-08-25 Published:2016-08-25

摘要/Abstract

摘要：

随着大数据的发展，Hadoop系统成为了大数据处理中的重要工具之一。在实际应用中，Hadoop的I/O操作制约系统性能的提升。通常Hadoop系统通过软件压缩数据来减少I/O操作，但是软件压缩速度较慢，因此使用硬件压缩加速器来替换软件压缩。Hadoop运行在Java虚拟机上，无法直接调用底层I/O硬件压缩加速器。通过实现Hadoop压缩器/解压缩器类和设计C++动态链接库来解决从Hadoop系统中获得压缩数据和将数据流向I/O硬件压缩加速器两个关键技术，从而将I/O硬件压缩加速器集成到Hadoop系统框架。实验结果表明，I/O硬件压缩加速器的每赫兹压缩速度为15.9 Byte/s/Hz，集成I/O硬件压缩加速器提升Hadoop系统性能2倍。

关键词: Hadoop, I/O, 硬件压缩加速器

Abstract:

With the development of big data, Hadoop systems become an important tool, but I/O operations impede their performance improvement in practical applications. Hadoop usually decreases its’ I/O operations by using software to compress data. However, data compression by software is slower than hardware accelerators. When Hadoop runs on Java virtual machines, it cannot directly call I/O hardware accelerators. To avoid getting data from the Hadoop system and transferring the data to I/O hardware accelerators, a compressor and decompressor class of Hadoop and a C++ dynamic linking library are employed in the Hadoop system. Experimental results show that both techniques can integrate I/O hardware accelerators into the Hadoop system frame work, the efficiency of I/O hardware compressor is 15.9Byte/s/Hz, and the performance of the Hadoop system can be improved by two times.

Key words: Hadoop, I/O, hardware compression accelerator

雷力1，钱斌海1，郭俊1，顾雄礼2，刘鹏1. 集成I/O硬件压缩加速器的Hadoop系统结构[J]. 计算机工程与科学.

LEI Li1,QIAN Bin-hai1,GUO Jun1,GU Xiong-li2,LIU Peng1. Integrated I/O hardware compression accelerators of Hadoop system architecture [J]. Computer Engineering & Science.

参考文献

［1］Hadoop ［EB/OL］. ［20150516］. http://hadoop.apache.org.
［2］MapReduce ［EB/OL］. ［20150516］. http://mareduce.apache.org.
［3］Ganglia［EB/OL］.［20150516］.http://ganglia.info/.
［4］TeraSort［EB/OL］. ［20150516］. http://www.highlyscalablesystems.com/3235/hadoopterasortbenchmark.
［5］Salomon D.Data compression: The complete reference［M］.New York:Springer Science & Business Media,2000.
［6］Putnam A,Caulfield A M,Chung E S,et al.A reconfigurable fabric for accelerating largescale datacenter services［J］.IEEE Micro,2015,35(3): 1022.
［7］Shan Y,Wang B,Yan J,et al.FPMR: MapReduce framework on FPGA［C］∥Proc of the 18th Annual ACM/SIGDA International Symposium on Field Programmable Gate Arrays (FPGA’10),2010:93102.
［8］Freund Y,Iyer R,Schapire R E,et al.An efficient boosting algorithm for combining preferences［J］.Journal of Machine Learning Research,2003(4):933969.
［9］Tsoi K H,Luk W.Axel: A heterogeneous cluster with FPGAs and GPUs［C］∥Proc of the 18th Annual ACM/SIGDA International Symposium on Field Programmable Gate Arrays (FPGA’10),2010:115124.
［10］Andrew M,Damir J,Kanak A.FPGAbased application acceleration: Case study with gzip compression/decompression streaming engine［C］∥Proc of International Conference on ComputerAided Design (ICCAD’13),2013:1.
［11］Abdelfattah M S, Hagiescu A, Singh D. Gzip on a Chip: High performance lossless data compression on FPGAs using OpenCL［C］∥Proc of the International Workshop on OpenCL 2013 & 2014 (IWOCL’14),2014:19.
［12］OpenCL for altera FPGAs: Accelerating performance and design productivity［EB/OL］.［20150516］.http://www.altera.com/products/software/opencl/openclindex.html.
［13］AHA3642［EB/OL］.［20150516］.http://www.aha.com/DrawProducts.aspx?Action=GetProductDetails & ProductID=38.

[1]	苏丽，孙彦猛，张博为，杨先博，朱颖. 一种基于Hadoop+CUDA实现相关器的方法[J]. J4, 20160101, 38(01): 46-51.
[2]	杨斌, 王敬宇, 刘世超, 邵明山, 肖伟, 陈起, 何晓斌, 刘卫国, 薛巍, . Beacon+：面向E级超级计算机的轻量级端到端I/O性能监控与分析诊断系统[J]. 计算机工程与科学, 2022, 44(09): 1521-1531.
[3]	葛旭冉, 刘洋, 陈志广, 肖侬. 基于MPI的并行大数据集生成器[J]. 计算机工程与科学, 2022, 44(07): 1152-1161.
[4]	徐海坤, 匡邓晖, 刘杰, 龚春叶, . 基于RMC的蒙特卡罗程序性能优化[J]. 计算机工程与科学, 2021, 43(04): 634-640.
[5]	赵俊生, 王鑫宇, 尹玉洁, 张林. 基于蒙古语新闻领域本体的分布式检索方法[J]. 计算机工程与科学, 2021, 43(03): 560-570.
[6]	杨青1,2,3，张亚文1,2，张琴1，袁佩玲1. 基于Hadoop的多维关联规则挖掘算法研究及应用[J]. 计算机工程与科学, 2019, 41(12): 2127-2133.
[7]	刘希明，李雨轩，宫晓利，张金. 虚拟化环境中线程级SLO保障的I/O软件框架研究[J]. 计算机工程与科学, 2019, 41(05): 773-779.
[8]	曹立强,罗红兵. 并行科学计算应用中采样数据的聚集I/O[J]. 计算机工程与科学, 2018, 40(09): 1534-1539.
[9]	王永坤1,罗萱1,金耀辉1,2. 基于私有云和物理机的混合型大数据平台设计及实现[J]. 计算机工程与科学, 2018, 40(02): 191-199.
[10]	刘鹏1,2，叶帅3，孟磊1,2，王灿4. 基于Spark的并行遗传算法求解多峰函数极值[J]. 计算机工程与科学, 2018, 40(02): 210-217.
[11]	肖文，胡娟，周晓峰. PFPonCanTree：一种基于MapReduce的并行频繁模式增量挖掘算法[J]. 计算机工程与科学, 2018, 40(01): 15-23.
[12]	蔡武越1,王珂2，郝玉洁2，段晓冉2. 一种Hadoop集群下的行为异常检测方法[J]. 计算机工程与科学, 2017, 39(12): 2185-2191.
[13]	皇甫永硕,刘杰,龚春叶. 基于二维结构化网格的可压缩流体并行算法研究[J]. 计算机工程与科学, 2017, 39(09): 1602-1609.
[14]	吴云蔚，宁芊. 基于Hadoop平台的分布式SVM参数寻优[J]. 计算机工程与科学, 2017, 39(06): 1042-1047.
[15]	黄伟建，杨海龙. Hadoop下改进布隆过滤器算法的网页去重[J]. 计算机工程与科学, 2017, 39(02): 285-290.