3 ms·
I’d believe half of all traffic to non-video hosts, however. We provide a lot of free content and it’s staggering how many bad robots there are - not actual mal
by acdha 4y ago
I’d believe half of all traffic to non-video hosts, however. We provide a lot of free content and it’s staggering how many bad robots there are - not actual malice, just what you’d expect from lousy programmers who don’t get the bill. Things like the same IP / User-Agent downloading the same file thousands of times just in case it changed in the last 5 seconds or crawling millions of permutations of search parameters rather than using the site maps. Many week that’s half of the total traffic, many tens of terabytes of HTML & JSON.
- Akronymus 4y agoAs someone who once wrote a scraper to interface with a site that lists lawyers (So we can blacklist them in advance from being called, because they REALLY don't like getting contacted by accident), and another time a scraper to get a list of unemployed people (employee leasing): Bad bots really ruin it for those that actually try to make well behaved scrapers.
- acdha 4y agoYes. Stuff like the default Python user agent being blocked by Cloudflare cause hassles for students & are entirely due to sloppy bot developers.