文件名称:nwebcrawlercSharp
- 所属分类：
- C#编程
- 资源属性：
- [Windows] [Visual.Net] [源码]
- 上传时间：
- 2012-11-26
- 文件大小：
- 2.02mb
- 下载次数：
- 0次
- 提 供 者：
- c*
- 相关连接：
- 无
- 下载说明：
- 别用迅雷下载，失败请重下，重下不扣分！
介绍说明－－下载内容均来自于网络，请自行研究使用
可配置：线程数、线程等待时间，连接超时时间，可爬取文件类型和优先级、下载目录等。
状态栏显示统计信息：排入队列URL数，已下载文件数，已下载总字节数，CPU使用率和可用内存等。
有偏好的爬虫：可针对爬取的资源类型设置不同的优先级。
健壮性：十几项URL正规化策略以排除冗余下载、爬虫陷阱避免策略的使用-c# crawler Threads
状态栏显示统计信息：排入队列URL数，已下载文件数，已下载总字节数，CPU使用率和可用内存等。
有偏好的爬虫：可针对爬取的资源类型设置不同的优先级。
健壮性：十几项URL正规化策略以排除冗余下载、爬虫陷阱避免策略的使用-c# crawler Threads
(系统自动生成,下载前可以参看下载内容)
下载文件列表
BuildProcessTemplates
.....................\DefaultTemplate.xaml
.....................\UpgradeTemplate.xaml
data
....\crawlerdb.s3db
....\pdc_09.txt
....\sina_12_28.txt
....\spider.ico
lib
...\x64
...\...\System.Data.SQLite.dll
...\...\System.Data.SQLite.lib
...\x86
...\...\System.Data.SQLite.dll
...\...\System.Data.SQLite.lib
setup
.....\build.cmd
.....\Files.wxs
.....\license.rtf
.....\Setup.wxs
.....\wix3
.....\....\candle.exe
.....\....\darice.cub
.....\....\light.exe
.....\....\wconsole.dll
.....\....\winterop.dll
.....\....\wix.dll
.....\....\WixUIExtension.dll
src
...\NWebCrawler.sln
...\NWebCrawlerLib
...\..............\Common
...\..............\......\FileSystemUtility.cs.cs
...\..............\......\Logger.cs
...\..............\......\PriorityQueue.cs
...\..............\......\SQLiteUtility.cs
...\..............\CrawlerThread.cs
...\..............\CrawlHistroyEntry.cs
...\..............\Downloader.cs
...\..............\EventStream.cs
...\..............\MemCache.cs
...\..............\NWebCrawlerLib.csproj
...\..............\PageRank.cs
...\..............\Parser.cs
...\..............\Program.cs
...\..............\Properties
...\..............\..........\AssemblyInfo.cs
...\..............\..........\Resource.en-US.resx
...\..............\..........\Resource.zh-CN.resx
...\..............\ResourceCulture.cs
...\..............\Settings.cs
...\..............\UrlQueueManager.cs
...\..............\Utility.cs
...\NWebCrawler
...\...........\app.config
...\...........\config.ini
...\...........\MainForm.cs
...\...........\MainForm.Designer.cs
...\...........\MainForm.resx
...\...........\NWebCrawler.csproj
...\...........\Program.cs
...\...........\Properties
...\...........\..........\AssemblyInfo.cs
...\...........\..........\Resources.Designer.cs
...\...........\..........\Resources.resx
...\...........\..........\Settings.Designer.cs
...\...........\..........\Settings.settings
...\...........\SettingsForm.cs
...\...........\SettingsForm.Designer.cs
...\...........\SettingsForm.resx
test
....\NWebCrawler.Test
....\................\NWebCrawler.Test.sln
....\................\NWebCrawler.Test
....\................\................\NormalizeTest.cs
....\................\................\NWebCrawler.Test.csproj
....\................\................\ParserTest.cs
....\................\................\Properties
....\................\................\..........\AssemblyInfo.cs
.....................\DefaultTemplate.xaml
.....................\UpgradeTemplate.xaml
data
....\crawlerdb.s3db
....\pdc_09.txt
....\sina_12_28.txt
....\spider.ico
lib
...\x64
...\...\System.Data.SQLite.dll
...\...\System.Data.SQLite.lib
...\x86
...\...\System.Data.SQLite.dll
...\...\System.Data.SQLite.lib
setup
.....\build.cmd
.....\Files.wxs
.....\license.rtf
.....\Setup.wxs
.....\wix3
.....\....\candle.exe
.....\....\darice.cub
.....\....\light.exe
.....\....\wconsole.dll
.....\....\winterop.dll
.....\....\wix.dll
.....\....\WixUIExtension.dll
src
...\NWebCrawler.sln
...\NWebCrawlerLib
...\..............\Common
...\..............\......\FileSystemUtility.cs.cs
...\..............\......\Logger.cs
...\..............\......\PriorityQueue.cs
...\..............\......\SQLiteUtility.cs
...\..............\CrawlerThread.cs
...\..............\CrawlHistroyEntry.cs
...\..............\Downloader.cs
...\..............\EventStream.cs
...\..............\MemCache.cs
...\..............\NWebCrawlerLib.csproj
...\..............\PageRank.cs
...\..............\Parser.cs
...\..............\Program.cs
...\..............\Properties
...\..............\..........\AssemblyInfo.cs
...\..............\..........\Resource.en-US.resx
...\..............\..........\Resource.zh-CN.resx
...\..............\ResourceCulture.cs
...\..............\Settings.cs
...\..............\UrlQueueManager.cs
...\..............\Utility.cs
...\NWebCrawler
...\...........\app.config
...\...........\config.ini
...\...........\MainForm.cs
...\...........\MainForm.Designer.cs
...\...........\MainForm.resx
...\...........\NWebCrawler.csproj
...\...........\Program.cs
...\...........\Properties
...\...........\..........\AssemblyInfo.cs
...\...........\..........\Resources.Designer.cs
...\...........\..........\Resources.resx
...\...........\..........\Settings.Designer.cs
...\...........\..........\Settings.settings
...\...........\SettingsForm.cs
...\...........\SettingsForm.Designer.cs
...\...........\SettingsForm.resx
test
....\NWebCrawler.Test
....\................\NWebCrawler.Test.sln
....\................\NWebCrawler.Test
....\................\................\NormalizeTest.cs
....\................\................\NWebCrawler.Test.csproj
....\................\................\ParserTest.cs
....\................\................\Properties
....\................\................\..........\AssemblyInfo.cs