Abstract:

With the exponential growth of network information resources and the growing personalized demands of customers, topicfocused crawler emerges as the times require. Topicfocused crawlers are programs designed to download web pages which are relevant to specific topics. Using information gathered at running time, topicfocused crawlers explore the webs which follow promissory hyperlinks, and fetch only pages which appear to be relevant. The searching engine and corpus building based on topicfocused crawling have been widely used. We first define the goals and operating principles of focused crawling, comprehensively analyze the recent advances at home and abroad, and then compare the crawling strategies of various topicfocused crawlers as well as the advantages and disadvantages of related algorithms. Finally, we point out the future direction of topicfocused crawling.

Key words: web crawler;focused-crawler;searching engine