网络爬虫，也叫网络蜘蛛，有的项目也把它称作“walker”。维基百科所给的定义是“一种系统地扫描互联网，以获取索引为目的的网络程序”。网络上有很多关于网络爬虫的开源项目，其中比较有名的是Heritrix和Apache Nutch。



有时需要在网上搜集信息，如果需要搜集的是获取方法单一而人工搜集费时费力的信息，比如统计一个网站每个月发了多少篇文章、用了哪些标签，为自然语言处理项目搜集语料，或者为模式识别项目搜集图片等等，就需要爬虫程序来完成这样的任务。而且搜索引擎必不可少的组件之一也是网络爬虫。 -Web crawler, also known as the spider web, some projects also called it walker . Wikipedia is defined as　a systematic scanning of the Internet, in order to obtain the index for the purpose of the network program . There are a lot of open source projects on the web crawler, which is more popular Apache and Nutch Heritrix.



Sometimes you need to collect information on the Internet, if you need to collect the method is a single and manual collection of information, such as a website each month made a number of articles, with which tags, for natural language processing project data collection, or for the pattern recognition project to collect pictures, and so on, you need to complete the task of crawler. And one of the essential components of the search engine is the web crawler.