Abstract:

Data analysis and processing is one of the most important tasks in largescale distributed data processing applications.Due to its simplicity and scalability,MapReduce programming model has gradually become the crucial model for largescale distributed data processing systems (eg.Hadoop).Since the data may be uniformly distributed,data skew occurs when MapReduce programming model joins data,thus degrading the join performance severely.To solve data skew,its reason is analyzed,the load balancing cost model is established,and the rangepartitioner algorithm is proposed to control data skew so as to realize load balancing.Experimental results demonstrate that our method can obviously improve the efficiency of joins.

Key words: MapReduce, join, data skew, rangepartitioner, load balancing