3 ms·
If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.
by mfontani 7y ago
If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.
- cft 7y agoBanning and kicking offline is different. I would love to sue them if say I attempted to build a search engine and they attacked me.
- andrerm 7y agoThat's not what they said. For the record, they said: > Identification of well known legitimate bots; > Hand written rules for simple bots that, however simple, get used day in, day out; > Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and > Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
- aussieguy1234 7y ago> Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is. Well, instead of using curl scripts, crawlers will switch to Selenium and use a real scripted browser