关键词: 大数据存储, 性能评测方法, 基准测试程序, 低效行为分析

Abstract: Big data storage plays an important role in the whole big data application framework system in view of the increasing amount of data. Performance evaluation for big data storage can guide big data application developers to analyze performance bottlenecks and optimize the performance of big data systems. In the past, benchmarking was usually used to evaluate the performance of different big data frameworks, or the performance analysis of distributed file systems was carried out by piling and analyzing track files. These two methods adopt different analytical perspectives, but there has not been a reasonable evaluation system to evaluate the distributed storage system of big data. This paper proposes the architecture and specific implementation of the big data storage performance evaluation method combining active and passive methods. In the active evaluation method, this paper provides benchmark test programs of more than 20 applications in 6 fields to initiate performance tests on big data storage systems, and analyzes the benchmark performance indicators of big data storage systems. In the passive performance evaluation method, this paper provides analysis and positioning methods for inefficient tasks, inefficient operators and inefficient functions. By analyzing the big data applications running on the big data storage systems, we can find out the reasons for the inefficiency of big data applications. Experiments show that the architecture of the proposed big data performance evaluation method can comprehensively evaluate the performance of big data storage.

Key words: big data storage, performance evaluation method, benchmark suite, inefficient behavior analysis