Abstract:

Face to the problems which exist in Web information mining the paper studies network crawler systems,and proposes a HTTP-based crawling method of in crement updating for reducing the network flow when a network crawler system runs. The method updates the current Web link database by the Web prefetch technique, and shows the effect close to the current network crawler systems when reducing the network flow.