Abstract:

The MapReduce model has provided strong support for the dataintensive supercomputing as a fundamental application flat. It has a singlenode task scheduler, which has a simple architecture and is convenient to control the worker nodes, while there exists the single node error problem. In Hadoop (Open Source MapReduce) released versions, it has three different mechanisms such as synchronization on demand, recovery from history logging and dropping. This paper analyses the data jam, result errors and efficiency decline in the three methods, and then gives a method for delivering the information of task dependencies to solve the problems.

Key words: MapReduce;Hadoop;task scheduling;single node error recovery;task dependency