crawl

How to avoid be baned

Anti-crawling strategy IP restrictions based on access frequency/quantity Real users and crawlers visit web site with different frequency. In general, crawlers have two features: most of the accesses are concentrated in a short time and accesses are time-periodic.