3 ms·
As someone who once wrote a scraper to interface with a site that lists lawyers (So we can blacklist them in advance from being called, because they REALLY don'
by Akronymus 3y ago
As someone who once wrote a scraper to interface with a site that lists lawyers (So we can blacklist them in advance from being called, because they REALLY don't like getting contacted by accident), and another time a scraper to get a list of unemployed people (employee leasing): Bad bots really ruin it for those that actually try to make well behaved scrapers.
- acdha 3y agoYes. Stuff like the default Python user agent being blocked by Cloudflare cause hassles for students & are entirely due to sloppy bot developers.