基于加速收敛蜂群算法的资源感知调度器

计算机工程与科学

基于加速收敛蜂群算法的资源感知调度器

江涛1，袁景凌1，陈旻骋1，宋华明2

（1.武汉理工大学计算机科学与技术学院,湖北武汉 430070;2.湖北省咸宁市公安局,湖北咸宁 437099）

收稿日期:2016-04-21 修回日期:2016-06-13 出版日期:2016-08-25 发布日期:2016-08-25
基金资助:
国家自然科学基金(61303029);教育部留学回国启动基金(［2012］1707);湖北省自然科学基金(2014CFB836)

Resource-aware scheduler based on bee colony algorithm with fast convergence

JIANG Tao1,YUAN Jing-ling1,CHEN Min-cheng1,SONG Hua-ming2

（1.School of Computer Science and Technology,Wuhan University of Technology,Wuhan 430070;
2.Xianning Public Security Bureau of Hubei Province,Xianning 437099,China）

Received:2016-04-21 Revised:2016-06-13 Online:2016-08-25 Published:2016-08-25

摘要/Abstract

摘要：

为了能有效处理海量数据，进行关联分析、商业预测等,Hadoop分布式云计算平台应运而生。但随着Hadoop的广泛应用，其作业调度方面的不足也显现出来，现有的多种作业调度器存在参数设置复杂、启动时间长等缺陷。借助于人工蜂群算法的自组织性强、收敛速度快的优势，设计并实现了能实时检测Hadoop内部资源使用情况的资源感知调度器。相比于原有的作业调度器，该调度器具有参数设置少、启动速度快等优势。基准测试结果表明，该调度器在异构集群上，调度资源密集型作业比原有调度器快10%～20%左右。

关键词: 云计算, Hadoop, 作业调度器, 人工蜂群, 资源感知

Abstract:

In order to effectively deal with massive data, correlation analysis, business forecast and so on, the distributed cloud computing platform Hadoop has emerged. But with the wide application of Hadoop, the job scheduling problems become prominent. Many existing job schedulers have defects such as complexity of parameters setting, long start time and so on. Taking full advantages of the artificial bee colony algorithm in terms of strong self-organizing ability and fast convergence, we design and implement a resource-aware scheduler of Hadoop which can detect real-time internal resources utilization. Compared to those classical job schedulers, the proposed scheduler has advantages of few parameters and fast starting speed. Experiments on benchmark programs show that the proposal can schedule resource intensive work faster than the original one by 10% ~ 20% on heterogeneous clusters.

Key words: cloud computing, Hadoop, job scheduler, artificial bee colony, resource-aware

江涛1，袁景凌1，陈旻骋1，宋华明2. 基于加速收敛蜂群算法的资源感知调度器[J]. 计算机工程与科学.

JIANG Tao1,YUAN Jing-ling1,CHEN Min-cheng1,SONG Hua-ming2. Resource-aware scheduler based on bee colony algorithm with fast convergence [J]. Computer Engineering & Science.

参考文献

［1］Xia Wei.Research and improvement of job scheduling algorithms in Hadoop platform ［D］.Guangzhou:South China University of Technology,2010.(in Chinese)
［2］Li Cheng-hua, Zhang Xin-fang, Jin Hai, et al. MapReduce:A new programming model for distributed parallel computing［J］.Computer Engineering & Science,2011,33(3):129-135.(in Chinese)
［3］Karaboga D,Basturk B.A powerful and efficient algorithm for numerical function optimization: Artificial bee colony (ABC) algorithm［J］.Journal of Global Optimization,2007,39(3):459-471.
［4］Zaharia M, Borthakur D, Sen Sarma J,et al.Delay scheduling:A simple technique for achieving locality and fairness in cluster scheduling［C］∥Proc of the 5th European Conference on Computer Systems,2010:1.
［5］Ji Peng-fei, Qi Jian-dong, Zhu Wen-fei.Application of improved artificial fish swarm algorithm in Hadoop scheduling algorithm［J］.Application Research of Computers,2014,31(12):3572-3575.(in Chinese)
［6］Liang Jing, Xu Bo, Ge Yu. Map-Reduce job scheduling algorithm based on improved shuffled frog leaping strategy［J］.Application Research of Computers,2013,30(7):1999-2002.(in Chinese)
［7］Zhang Q,Zhani M F,Yang Y,et al.PRISM: Fine-grained resource-aware scheduling for MapReduce［J］.IEEE Transactions on Cloud Computing,2015,3(2):1.
［8］Mude R G, Betta A, Debbarma A. Capturing node resource status and classifying workload for map reduce resource aware scheduler［M］∥Intelligent Computing,Communication and Devices.India:Springer,2015:247-257.
［9］Hu W,Tian C,Liu X,et al.Multiple-job optimization in MapReduce for heterogeneous workloads［C］∥Proc of the 6th International Conference on Semantics Knowledge and Grid.2010:135-140.
［10］Zhang Hai-yan, Wu Fan, Wang Jian-xin. Research of Hadoop resource-aware scheduler based on ant colony algorithm［J］.Computer Engineering and Applications,2014,50（15）：65-71.(in Chinese)
［11］Xu Xiao-fei,Liu Zhi-zhong,Wang Zhong-jie,et al.S-ABC—Service domain-oriented artificial bee colony algorithm paradigm［J］.Chinese Journal of Computers,2015,38(11):2301-2317.(in Chinese)
［12］Karaboga N.A new design method based on artificial bee colony algorithm for digital IIR filters［J］.Journal of the Franklin Institute,2009,346(4):328-348.
［13］Bi Xiao-jun, Wang Yan-jiao. Artificial bee colony algorithm with fast convergence［J］.Systems Engineering & Electronics,2011,33(12):2755-2761.(in Chinese)
附中文参考文献：
［1］夏祎.Hadoop平台下的作业调度算法研究与改进［D］.广州：华南理工大学，2010.
［2］李成华,张新访,金海,等.MapReduce:新型的分布式并行计算编程模型［J］.计算机工程与科学,2011,33(3):129-135.
［5］吉鹏飞,齐建东,朱文飞.改进人工鱼群算法在Hadoop作业调度算法的应用［J］.计算机应用研究,2014,31(12):3572-3575.
［6］梁静,许波,葛宇.基于改进蛙跳策略的Map-Reduce作业调度算法［J］.计算机应用研究,2013,30(7):1999-2002.
［10］张海燕,吴凡,王建新.基于蚁群算法的Hadoop资源感知调度器研究［J］.计算机工程与应用,2014,50(15):65-71.
［11］徐晓飞,刘志中,王忠杰,等.S-ABC--面向服务领域的人工蜂群算法范型［J］.计算机学报,2015,38(11):2301-2317.
［13］毕晓君,王艳娇.加速收敛的人工蜂群算法［J］.系统工程与电子技术,2011,33(12): 2755-2761.

[1]	苏丽，孙彦猛，张博为，杨先博，朱颖. 一种基于Hadoop+CUDA实现相关器的方法[J]. J4, 20160101, 38(01): 46-51.
[2]	李卫平1,2，武海燕2，杨杰1. 基于效益博弈的云计算资源动态可协调分配策略研究[J]. J4, 20160101, 38(01): 57-61.
[3]	杨柏蔼, 赵山, 刘芳. 无服务器计算技术研究综述[J]. 计算机工程与科学, 2022, 44(04): 611-619.
[4]	庄鹤林, 杨火根, 夏小云, 廖伟志. 关于矩阵乘法问题的人工蜂群优化算法研究[J]. 计算机工程与科学, 2021, 43(12): 2131-2138.
[5]	赵俊生, 王鑫宇, 尹玉洁, 张林. 基于蒙古语新闻领域本体的分布式检索方法[J]. 计算机工程与科学, 2021, 43(03): 560-570.
[6]	李波, 侯鹏, 牛力, 武浩, 丁洪伟. 基于软件定义网络的云边协同架构研究综述[J]. 计算机工程与科学, 2021, 43(02): 242-257.
[7]	廖建锦, 孙庆骁, 杨海龙, 栾钟治, 钱德沛. 满足工作流执行时限的可抢占虚拟机实例配置和调度方法研究[J]. 计算机工程与科学, 2020, 42(11): 1956-1964.
[8]	林涛, 冯竞凯, 郝章肖, 黄少群. 基于组合预测模型的云计算资源负载预测研究[J]. 计算机工程与科学, 2020, 42(07): 1168-1173.
[9]	包敏泽, 胡秀婷, 谢玉莹, 蒋波. 基于人工蜂群算法的p-center问题求解算法[J]. 计算机工程与科学, 2020, 42(06): 1127-1133.
[10]	赵凤1,2，孔令润1,2，马改妮1,2. 多目标粒子群和人工蜂群混合优化的阈值图像分割算法[J]. 计算机工程与科学, 2020, 42(02): 281-290.
[11]	杨青1,2,3，张亚文1,2，张琴1，袁佩玲1. 基于Hadoop的多维关联规则挖掘算法研究及应用[J]. 计算机工程与科学, 2019, 41(12): 2127-2133.
[12]	史文杰，詹雨奇，李奎. 一种面向多平台航电资源建模及其调度算法[J]. 计算机工程与科学, 2019, 41(11): 1976-1984.
[13]	刘梓璇，周建涛. 负载均衡的主导资源公平分配算法[J]. 计算机工程与科学, 2019, 41(09): 1574-1580.
[14]	曹云梦1,2,周胜军3,刘晨1,2,韩燕波1,2. 面向服务的云-端动态协作方法[J]. 计算机工程与科学, 2019, 41(04): 598-605.
[15]	常小刚1，赵红星2. 动态调节因子的邻域搜索人工蜂群算法[J]. 计算机工程与科学, 2019, 41(04): 673-681.