基于运行数据分析的Spark任务参数优化

J4 ›› 2016, Vol. 38 ›› Issue (01): 11-19.

基于运行数据分析的Spark任务参数优化

陈侨安1，李峰1，曹越1，龙明盛1,2

(1.清华大学软件学院,北京 100084;2.清华大学信息科学与技术国家实验室（筹）,北京 100084)

收稿日期:2015-10-02 修回日期:2015-12-10 出版日期:2016-01-25 发布日期:2016-01-25
基金资助:
清华大学信息科学与技术国家实验室大数据科学与技术专项(面向领域的大数据应用系统开发与运行平台)

Parameter optimization for Spark jobs based
on runtime data analysis

CHEN Qiaoan1,LI Feng1,CAO Yue1,LONG Mingsheng1,2

(1.School of Software,Tsinghua University,Beijing 100084;2.National Laboratory for Information Science and Technology （TNList）,Tsinghua University,Beijing 100084,China)

Received:2015-10-02 Revised:2015-12-10 Online:2016-01-25 Published:2016-01-25

摘要/Abstract

摘要：

运行数据是大数据系统中增长最快、最为复杂也是最有价值的数据资源之一。基于运行数据，软件开发者可以分析关于软件质量和开发模型的重要信息。Spark作为一个分布式系统，在运行过程中会产生大量的运行数据，包括日志数据、监控数据以及任务图数据。开发者可以基于运行数据对系统进行参数调优。然而该系统所涉及的参数种类繁多、影响多样且难以评估，若对系统了解不足，进行参数调优存在较大的困难。提出运行数据历史库的概念，历史库中存储的是以往运行任务的特征信息以及运行配置信息。同时提出了基于历史库搜索的参数优化模型，并实验验证了本文提出的参数优化模型对用户任务性能提升具有较好的效果。

关键词: 大数据, 运行数据, 数据分析, 参数优化, Spark

Abstract:

The fast growing runtime data is one of the most complicated and valuable data resources in big data systems. Based on runtime data, developers can analyze software quality and discover important information on software development model. As a distributed system, Spark generates a large amount of runtime data during running user applications. Those runtime data include log data, monitoring data and graph representation of jobs. Developers can optimize system parameters with the help of runtime data. However, there are different types of parameters in Spark and it is difficult to identify the effects of the parameters, which makes them hard to tune. In this paper we propose the concept of runtime data historical database and a parameters optimization model based on searching the database. Experimental results validate that the proposed optimization model achieves good performance on the recommendation of system parameters.

Key words: big data;runtime data;data analysis;parameters optimization;Spark

陈侨安1，李峰1，曹越1，龙明盛1,2. 基于运行数据分析的Spark任务参数优化[J]. J4, 2016, 38(01): 11-19.

CHEN Qiaoan1,LI Feng1,CAO Yue1,LONG Mingsheng1,2. Parameter optimization for Spark jobs based
on runtime data analysis [J]. J4, 2016, 38(01): 11-19.

参考文献［17］

［1］	White T. Hadoop:The definitive guide［J］. Oreilly Media Inc Gravenstein Highway North,2010,215（11）:14．
［2］	Lakshman A,Malik P. Cassandra:A decentralized structured storage system［J］. Acm Sigops Operating Systems Review,2010,44（2）:3540．
［3］	Zaharia M,Chowdhury M,Franklin M J,et al. Spark:Cluster computing with working sets［C］∥Proc of the 2nd USENIX Conference on Hot Topics in Cloud Computing,2010:17651773．
［4］	Seo S,Jang I,Woo K,et al. HPMR:Prefetching and preshuffling in shared MapReduce computation environment［C］∥Proc of the 2009 IEEE International Conference on Cluster Computing,2009:18．
［5］	Jiang D,Ooi B C,Shi L,et al. The performance of MapReduce:An indepth study［J］. Proceedings of the VLDB Endowment,2010,3（12）:472483．
［6］	Dittrich J. Hadoop++:Making a yellow elephant run like a cheetah （without it even noticing）［J］. Proceedings of the VLDB Endowment,2010,3（12）:518529．
［7］	Shivnath B. Towards automatic optimization of MapReduce programs［C］∥Proc of the 1st ACM Symposium on Cloud Computing,2010:137142．
［8］	Herodotou H,Lim H, Luo G, et al. Starfish:A selftuning system for big data analytics［C］∥Proc of the 5th Cidr Conf,2011:261272．
［9］	Shi Juwei,Zhou Jia, Lu Jiaheng, et al. MRTuner:A toolkit to enable holistic optimization for MapReduce jobs［C］∥Proc of the VLDB Endowment, 2014,7(13):13191330．
［10］	Aaron D,Andrew O.Optimizing shuffle performance in spark［R］.CA:BerkeleyDepartment of Electrical Engineering and Computer Sciences,University of California,2013．
［11］	Ravi N. Configuring and optimizing spark applications with easeNishkam ravi,Cloudera［EB/OL］.［20150901］.https://apachebigdata2015.sched.org/event/55afa6d65370a56bdbcb5eba5166f010#.VemuzvaqpEN．
［12］	Lin J,Keogh E,Wei L,et al. Experiencing SAX:A novel symbolic representation of time series［J］. Data Mining & Knowledge Discovery,2007,15（2）:107144．
［13］	Xu W,Huang L,Fox A,et al. Detecting largescale system problems by mining console logs［C］∥Proc of the 27nd International Conference on Machine Learning,2010.DOI:10.1145/1629575.1629587．
［14］	Shieh J,Keogh E. iSAX:indexing and mining terabyte sized time series［C］∥Proc of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ACM,2008:623631．
［15］	Zaharia M,Chowdhury M,Das T,et al. Resilient distributed datasets:A faulttolerant abstraction for inmemory cluster computing［C］∥Proc of the 9th USENIX Conference on Networked Systems Design and Implementation, 2011:141146．
［16］	Dijkman R, Dumas M, GarciaBanuelos. Graph matching algorithms for business process model similarity search［C］∥Proc of the 7th International Conference on Business Process Management,2009:4863．
［17］	Bottou L. Largescale machine learning with stochastic gradient descent［C］∥Proc of COMPSTAT’2010,2010:177186．

[1]	陈侨安1，李峰1，曹越1，龙明盛1,2. 基于运行数据分析的Spark任务参数优化[J]. J4, 20160101, 38(01): 11-19.
[2]	王星苏, 熊文, 张瑞. 海量地铁乘客轨迹相似性连接方法：以深圳地铁为例[J]. 计算机工程与科学, 2023, 45(08): 1383-1392.
[3]	杨浩艺, 陈微, 姚泽欢, 谭郁松, 李非. 基于转录组学数据的抗真菌药物预测方法研究[J]. 计算机工程与科学, 2023, 45(02): 246-251.
[4]	胡艳芳, 熊文, 高炜. 基于 Spark 平台的网络游戏用户流失预测方法[J]. 计算机工程与科学, 2022, 44(10): 1730-1737.
[5]	葛旭冉, 刘洋, 陈志广, 肖侬. 基于MPI的并行大数据集生成器[J]. 计算机工程与科学, 2022, 44(07): 1152-1161.
[6]	刘世缘, 李云春, 陈晨, 杨海龙. 面向大数据存储的主动与被动相结合的性能评测方法体系结构与实现[J]. 计算机工程与科学, 2022, 44(04): 584-593.
[7]	杨柏蔼, 赵山, 刘芳. 无服务器计算技术研究综述[J]. 计算机工程与科学, 2022, 44(04): 611-619.
[8]	卞琛, 修位蓉, 于炯. 异构Spark集群数据倾斜修正调度策略[J]. 计算机工程与科学, 2022, 44(04): 620-630.
[9]	吕高锋, 王玉鹏, 杨鎔嘉, 唐竹. 基于聚合的FlowRadar网络数据采集加速模型设计[J]. 计算机工程与科学, 2022, 44(02): 220-226.
[10]	张元鸣, 虞家睿, 陆佳炜, 高飞, 肖刚. 基于Spark Streaming的视频大数据并行处理方法[J]. 计算机工程与科学, 2021, 43(10): 1736-1743.
[11]	李静, 莫思敏 . 基于改进遗传算法的深度神经网络优化研究[J]. 计算机工程与科学, 2021, 43(08): 1503-1511.
[12]	黄山, 房六一, 徐浩桐, 段晓东, . 面向容器环境的Flink的任务调度优化研究[J]. 计算机工程与科学, 2021, 43(07): 1173-1184.
[13]	刘亚波, 吴秋轩. 基于长短时记忆网络的电商大数据同一性标定[J]. 计算机工程与科学, 2021, 43(03): 407-415.
[14]	李琼, 宋振龙, 袁远, 谢徐超. 一种基于NVMeoF存储池的分域共享并发存储架构[J]. 计算机工程与科学, 2020, 42(10高性能专刊): 1711-1719.
[15]	赵奇奇. “魔方-3”高性能计算机运维管理平台设计与实现[J]. 计算机工程与科学, 2020, 42(10高性能专刊): 1807-1814.