Abstract:

In order to effectively deal with massive data, correlation analysis, business forecast and so on, the distributed cloud computing platform Hadoop has emerged. But with the wide application of Hadoop, the job scheduling problems become prominent. Many existing job schedulers have defects such as complexity of parameters setting, long start time and so on. Taking full advantages of the artificial bee colony algorithm in terms of strong self-organizing ability and fast convergence, we design and implement a resource-aware scheduler of Hadoop which can detect real-time internal resources utilization. Compared to those classical job schedulers, the proposed scheduler has advantages of few parameters and fast starting speed. Experiments on benchmark programs show that the proposal can schedule resource intensive work faster than the original one by 10% ~ 20% on heterogeneous clusters.

Key words: cloud computing, Hadoop, job scheduler, artificial bee colony, resource-aware