Abstract:

Face to the problems which exist in Web information mining the paper studies network crawler systems,and proposes a HTTP-based crawling method of in crement updating for reducing the network flow when a network crawler system runs. The method updates the current Web link database by the Web prefetch technique, and shows the effect close to the current network crawler systems when r educing the network flow.

Key words: information retrieval, web crawler, increment updating