Abstract:

The Genetic Algorithm (GA) needs many computation iterations in solving multimodal function extremums, so its running efficiency is too low when dealing with large-scale data, which greatly limits its practical application. The classical parallel platform Hadoop can improve the GA running efficiency to some extent, while the state-of-the-art parallel platform Spark can release much more parallelism of GA by realizing parallel crossover, mutation and other operations on each computing node. For the convenience of comparison, the GA solving multimodal function extremums are designed and implemented on single node, Hadoop and Spark, respectively. Experimental results show that, compared with single node platform and Hadoop platform, the Spark based implementation not only significantly reduces the running time but also effectively avoids the problem of premature convergence because of its powerful randomness, while dealing with large-scale samples.

Key words: genetic algorithm, multimodal function, extremum, parallel computing, Spark, Hadoop