Abstract:

Nowadays, the development of disk I/O never catches up with CPU according to the Moore’s law, and Network I/O is scarce, so I/O often becomes a bottleneck of data processing. Hadoop can store PBlevel data where I/O problem becomes more obvious. Compression is an important method to optimize I/O, which can reduce I/O load and speed up data transmission on disk and network. In Hadoop, the benefits of using compression have not been completely exploited. In this paper we first analyze the compression algorithms supported by Hadoop, then propose a strategy to help Hadoop users identify how to use compression and how to verify through experiment. By using compression, the performance of some Hadoop applications can be improved up to 65%.

Key words: Hadoop;MapReduce;I/O;compression