关键词: 聚焦爬虫, 爬行控制器, 主题相关度, 数据抽取

Abstract: The paper presents the design method for a focused crawler based on the crawling controller and the page analysis filter. Starting from the theme to be retrieved, the method based on the improved genetic algorithm combines with the advantages of both content evaluation and link structure. The crawler regards the URL link as the genetic individual,and the topicwordsbased VSM is applied to assess individual fitness, and imports new URLs to achieve crossover and mutation operations, and the URLs that have the same prefix are regarded as niche. The experimental results show that the approach has better performance.

Key words: focused crawler;crawling controller;topic relevancy;data extraction