关键词: Web日志预处理, 会话识别, MapReduce, 分布式处理

Abstract:

Web log preprocessing attracts more and more attention due to the importance of its output result. Meanwhile distributed processing of massive data based on Hadoop is being widely studied and applied, so Web log preprocessing with MapReduce becomes an inevitable development trend. In order to improve the accuracy of session identification results, we propose a new method to identify user session based on network topology and dynamic threshold. The current research state is analyzed and the advantages of this method are also discussed. Then, the MapReduce model is used to implement the distributed processing of the new method. Experimental results demonstrate high efficiency and high accuracy of the proposed method.

Key words: Web log preprocessing;session identification;MapReduce;distributed processing