关键词: Flink, 容器, 通信开销, 负载均衡, 迁移

Abstract: As a new generation of big data computing engine, Flink has been widely used. When containers of Flink are deployed in cloud environment, its default task scheduling algorithm cannot perceive node resources information and adjust the load in time, and the capacity for independent equilibrium is poorer. Although mainstream container layout tools provide the possibility of container management, they fails to combine Flink characteristics to solve the problem of balancing the resource utilization while reducing the communication overhead in the container group. Aiming at the above problem, this paper proposes a Flink load balancing strategy for cloud environment, which comprehensively considers the distribution characteristics of operators in Flink cluster and the communication mechanism between containers, and takes the communication cost between nodes and load balancing as evaluation criteria. Experimental results show that, compared with Flink default scheduling algorithm, this algorithm can effectively improve the computing efficiency and system performance.

Key words: Flink, container, communication overhead, load balancing, migration