Abstract:

The processing speed of Spark which is a big data processing structure is highly influenced by resource scheduling modes and whether we can utilize the resource sufficiently. Taking memories and the number of free cores into consideration, we propose a new scalable resource scheduling method. In this method, we monitor the resource utilization of nodes in real time and examine CPU processing speed and CPU residual utilization. This method can be used to optimize Spark Web service so as to meet the requirements of high concurrent request and low latency response and efficiently reduce the imbalance of resource utilization, thus improving resource utilization. Experimental results show that our method can obtain better results.

Key words: Spark, Web service, resource monitoring, resource scheduling