Abstract:

To solve the space waste problem existing in the server space where a lot of duplicated and similar data are stored, we propose an improved Bloom Filter algorithm, which adds an array of bit and dynamically optimizes the number of copies of duplicated data according to the weight calculated by the repeated hits of the bit array. Then, the improved algorithm is  parallelized in the Hadoop distributed cluster to further improve the processing efficiency. Experimental results show that compared with traditional web duplicate removal algorithms, the improved Bloom filter algorithm can not only improve the processing efficiency of jobs, but also save the server storage space to a certain extent by dynamically optimizing the number of copies of duplicated data according to the repeated hits of the bit array.

Key words: Hadoop, Bloom Filter, number of copy, MapReduce