A data skew correction scheduling strategy of heterogeneous Spark cluster
BIAN Chen1，XIU Wei-rong2，YU Jiong3
(1.College of Internet Finance and Information Engineering,Guangdong University of Finance,Guangzhou 510521;
2.School of Information Technology and Engineering,Guangzhou College of Commerce,Guangzhou 511363;
3.College of Information Science and Engineering,Xinjiang University,Urumqi 830046,China)Abstract:Due to the barrel effect of heterogeneous Spark clusters, unreasonable parallelism leads to poor adaptation of task allocation to the power of workers, which affects cluster computational efficiency and resource utilization. Aiming at this issue, the node resource models are firstly established, the coupling relationships among data distribution, parallelism parameters and task allocation are analyzed, and the optimization objective of the algorithm is proposed. The data skew correction scheduling strategy DSCS for heterogeneous Spark cluster is designed, which includes the parallelism prediction algorithm, data skew correction algorithm and heterogeneous task allocation algorithm. The prediction algorithm sets the parallelism degree in advance, the data skew correction algorithm performs data re-partitioning and parallelism correction according to the statistical information of the first stage, and the heterogeneous task allocation algorithm reasonably allocates the tasks to the workers according to their computing capabilities in the heterogeneous cluster, so as to improve the adaptability of data volume and workers power and optimize the overall performance of Spark cluster. The experimental results show that the algorithm achieves performance improvement under different job types and dataset conditions, and can effectively reduce the probability of spill to the external storage of workers.
