spider

使用 Scrapoxy 构建自己的爬虫代理池

简介 现在如今爬虫的使用场景随处可见,从搜索引擎构建内容索引到

Scrapy

Scrapy Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

How to avoid be baned

Anti-crawling strategy IP restrictions based on access frequency/quantity Real users and crawlers visit web site with different frequency. In general, crawlers have two features: most of the accesses are concentrated in a short time and accesses are time-periodic.