Abstract:

With the development of distributed computing technology, Hadoop, as a typical representative in the field of massive data processing, is vulnerable to hidden security threats, such as data breaches, due to weak security mechanism and lack of user activity monitoring. By combining with the characteristics of the principal component analysis, we perform parallel process through MapReduce to overcome the disadvantage of principal component analysis and improve the training efficiency. We propose an abnormal behavior detection method in Hadoop cluster, namely we compare the current user behavior patterns with historical behavior patterns to see if they match, which is taken as a metric for anomaly behavior detection. Experimental results indicate that our method can detect users' anomaly behavior effectively.

Key words: Hadoop cluster, principal component analysis, anomaly detection, MapReduce, behavior pattern