3 ms·
Using the law to block crawlers is more like saying: 1. Google can come in 2. Other Americans can't come in 3. Chinese people can come in (or anywhere else w
by gakada 10y ago
Using the law to block crawlers is more like saying:
1. Google can come in
2. Other Americans can't come in
3. Chinese people can come in (or anywhere else where US laws don't apply)
It might not be unfair, but it is certainly pointless and arbitrary.
- jessedhillon 10y agoTo be fair, many companies which take anti-scraping seriously will also take inputs like geographic origin of a request into consideration when applying request throttling and filtering.