关键词: 数据质量, 数据清洗, 相似重复记录, SNM算法

Abstract: The problematic data in the data warehouse has a great impact on data quality. In order to find and delete these problematic data, the primary work is the processing of similar repeated data. Currently, the most widely used algorithm for deduplication is the sorted-neighborhood method (SNM). After  analyzing the shortcomings of this algorithm, an improved SNM algorithm (ISNM) is proposed. The attribute weights are calculated using the attribute discrimination method, which solves the subjectivity caused by artificial weights. The field filtering algorithm is used to calculate the similarity of two records, which reduces the number of comparisons of record attributes in the window and accelerates the detection speed of the algorithm. Variable windows are used instead of fixed-size windows to prevent missing records and reduce useless record comparisons. Experimental results show that ISNM algorithm has obvious advantages in terms of recall, precision and running time overhead.

Key words: data quality, data cleaning, similar duplicate records, SNM algorithm