10 ms·

I used to work for a data-scraping firm and very often we would accidentally knock many web sites offline when we pointed our crawlers at them. I'd love to agr

by tomc1985 7y ago

I used to work for a data-scraping firm and very often we would accidentally knock many web sites offline when we pointed our crawlers at them. I'd love to agree with you, but the crawler problem is 100x worse today than it was a decade ago

deogeo 7y ago
This would be much better solved with IP-based rate limits. And if IP-based doesn't work, then you're dealing with a DDOS, and it doesn't sound like this case was DDOS protection.
- tomc1985 7y ago
  IP-based rate limiting is easily foiled via proxies, VPN services, tor, or botnets
  - deogeo 7y ago
    And user agent string based protections are even more easily foiled, that's why I don't believe this can be plausibly counted under DDoS protection.