关键词: Hadoop, MapReduce, 批量作业, 调度优化, 最小化总完工时间

Abstract:

As large-scale MapReduce clusters become widely adapted to process huge amount of data, one of critical challenges is to improve the service quality of MapReduce clusters by minimizing their makespan. A scheduling model can be considered for multiple MapReduce jobs. It is observed that the order in which these jobs are executed can have a significant impact on their overall makespan. The goal of the paper is to design a framework of automatic job scheduler and propose an analytical model for minimizing the makespan of such a set of MapReduce jobs. By considering a better strategy and implementation, we can meet the conditions of the classical Johnson algorithm and use it to find the optimal solution. Under our proposed new strategy, solving the balanced pools problem becomes exact in linear time, better than existing simulating approaches. Our proposed analytical results can be applied to improve system response time, energyefficiency and load-balance in Hadoop cluster pools, while corresponding numerical examples validate our observations.

Key words: Hadoop;MapReduce;batch workloads;optimized schedule;minimized makespan