数据流Eager传输：一种分布式流体系结构中的性能优化技术

J4 ›› 2015, Vol. 37 ›› Issue (11): 2035-2044.

数据流Eager传输：一种分布式流体系结构中的性能优化技术

李鑫1, 3，郭晓威1，林宇斐2

(1.国防科学技术大学高性能计算国家重点实验室，湖南长沙 410073；
2.国防科学技术大学研究生院，湖南长沙 410073；3.总参第六十三研究所，江苏南京 210007)

收稿日期:2015-08-11 修回日期:2015-10-15 出版日期:2015-11-25 发布日期:2015-11-25
基金资助:
国家自然科学基金资助项目（61221491，61303071）

Stream Eager transmission:a performance optimization
technique for the distributed stream architecture

LI Xin1,3,GUO Xiaowei1,LIN Yufei2

(1.State Key Laboratory of High Performance Computing,National University of Defense Technology,Changsha 410073;
2.Graduate School,National University of Defense Technology,Changsha 410073;
3.The 63rd Research Institute of PLA General Staff Headquarters,Nanjing 210007,China)

Received:2015-08-11 Revised:2015-10-15 Online:2015-11-25 Published:2015-11-25

摘要/Abstract

摘要：

分布式流体系结构扩展了分布式环境下的流计算模型, 可在互联网上为大数据计算应用提供高效低成本的运行环境。互联网中较长的通信开销制约了计算性能。提出了一种数据流Eager传输的性能优化技术，以挖掘计算与通信之间的并行性，隐藏通信延迟。在分布式流体系结构原型系统中实现了该技术。实验结果表明，应用程序采用该优化技术之后的平均时间开销减少了19.58%，表明该优化技术能够显著提高应用的性能，具有良好的应用前景。

关键词: 分布式流体系结构, 性能优化, 数据流Eager传输技术

Abstract:

The distributed stream architecture extends the stream model in the distributed environment, which can provide an efficient and lowcost execution environment for big data computing applications in the Internet. Meanwhile, the long communication overhead in the Internet restricts the performance of the distributed stream architecture. We propose a data stream Eager transmission technique and implement it in a prototype system of the distributed stream architecture, which improves the performance of applications by exploring the parallelism between communications and computation, as well as hides communication latency. Experimental results show that the proposed technique averagely gains a 19.58% reduction of the execution time cost, significantly improves the performance of the applications and has a promising prospec.

Key words: distributed stream architecture;performance optimization;stream Eager transmission technique

李鑫1, 3，郭晓威1，林宇斐2. 数据流Eager传输：一种分布式流体系结构中的性能优化技术[J]. J4, 2015, 37(11): 2035-2044.

LI Xin1,3,GUO Xiaowei1,LIN Yufei2. Stream Eager transmission:a performance optimization
technique for the distributed stream architecture [J]. J4, 2015, 37(11): 2035-2044.

参考文献

［1］Mell P, Grance T. The NIST definition of cloud computing ［R］. NIST, 2011.
［2］Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters ［J］. Communications of the ACM, 2008, 51(1): 107113.
［3］Zaharia M, Das T, Li H, et al. Discretized streams: Faulttolerant streaming computation at scale ［C］∥Proc of the 24th ACM Symposium on Operating Systems Principles, 2013: 423438.
［4］Xin R S, Rosen J, Zaharia M, et al. Shark: SQL and rich analytics at scale［C］∥Proc of the 2013 ACM SIGMOD International Conference on Management of DataACM, 2013:1324.
［5］Thusoo A, Sarma J S, Jain N, et al. Hivea petabyte scale data warehouse using Hadoop［C］∥Proc of IEEE 26th International Conference on Data Engineering, 2010:9961005.
［6］Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: A faulttolerant abstraction for inMemory cluster computing ［C］∥Proc of the 9th USENIX Conference on Networked Systems Design and Implementation, 2012:2.
［7］Zaharia M, Chowdhury M, Franklin M J, et al. Spark: Cluster computing with working sets ［C］∥Proc of the 2nd USENIX Conference on Hot Topics in Cloud Computing, 2010:10.
［8］Foster I, Kesselman C. The Grid 2: Blueprint for a new computing infrastructure ［M］. New York:Elsevier, 2003.
［9］Chawathe Y, Ratnasamy S, Breslau L, et al. Making gnutellalike P2P systems scalable［C］∥Proc of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, 2003:407418.
［10］Jeffers J, Reinders J. Intel Xeon Phi coprocessor highperformance programming［J］. San Francisco:Morgan Kaufmann, 2013.
［11］Xie M, Lu Y, Liu L, et al. Implementation and evaluation of network interface and message passing services for TianHe1A supercomputer［C］∥Proc of the 2011 19th Annual IEEE Symposium on High Performance Interconnects, 2011:7886.
［12］Xue W, Yang C, Fu H, et al. Enabling and scaling a global shallowwater atmospheric model on Tianhe2［C］∥Proc of the 2014 IEEE 28th International Parallel and Distributed Processing Symposium, 2014:745754.

[1]	康宇晗, 时洋, 陈照云, 文梅. 面向迈创+MatrixZone异构系统的深度学习编程框架[J]. 计算机工程与科学, 2023, 45(07): 1149-1158.
[2]	莫舒恒, 卢圣有, 黄聃, 卢宇彤. 基于即时编译的GNU Octave性能优化[J]. 计算机工程与科学, 2022, 44(12): 2091-2101.
[3]	沈佳杰, 卢修文, 向望, 赵泽宇, 王新, . 分布式存储系统读写一致性算法性能优化研究综述[J]. 计算机工程与科学, 2022, 44(04): 571-583.
[4]	张驭洲, 曹武迪, 卜景德, 谭光明, 吉青. GROMACS 2020在ROCm平台上的移植与优化[J]. 计算机工程与科学, 2021, 43(11): 1901-1909.
[5]	周静, 关玉蓉. 基于SDN的DWSN技术分析及性能优化研究[J]. 计算机工程与科学, 2021, 43(08): 1413-1421.
[6]	朱良杰, 沈佳杰, 周扬帆, 王新, . 云际存储系统性能优化研究现状与展望[J]. 计算机工程与科学, 2021, 43(05): 761-772.
[7]	徐海坤, 匡邓晖, 刘杰, 龚春叶, . 基于RMC的蒙特卡罗程序性能优化[J]. 计算机工程与科学, 2021, 43(04): 634-640.
[8]	王武1，王舒扬1,2，姜金荣1,孟虹松3. 快速多极子方法在申威众核处理器上的实现和优化[J]. 计算机工程与科学, 2019, 41(07): 1161-1167.
[9]	曹立强,罗红兵. 并行科学计算应用中采样数据的聚集I/O[J]. 计算机工程与科学, 2018, 40(09): 1534-1539.
[10]	廖旺坚1,2,黄永峰1,2,包从开1,2. Spark并行计算框架的内存优化[J]. 计算机工程与科学, 2018, 40(04): 587-593.
[11]	李帅1，吴斌2，杜修明3，陈玉峰3. 基于Spark的BIRCH算法并行化的设计与实现[J]. 计算机工程与科学, 2017, 39(01): 35-41.
[12]	李鑫1,3,4，林宇斐2，郭晓威1. 面向分布式流体系结构的多副本积极容错技术[J]. J4, 2015, 37(12): 2233-2241.
[13]	陆平静，李宝，易任娇，张英，王绍刚，庞征斌. 一种基于改进模拟退火算法的程序性能优化参数搜索算法[J]. J4, 2015, 37(07): 1227-1232.
[14]	熊敏，王勇献. 面向CPU+MIC混合异构平台的地震波叠前时间偏移算法并行与优化策略[J]. J4, 2015, 37(01): 14-22.
[15]	迟利华，胡庆丰，刘杰，甘新标，蒋杰，晏益慧. 面向FT1000微处理器的STREAM并行计算与优化[J]. J4, 2014, 36(12): 2267-2271.