3 ms·
And only a few paragraphs down from a heading that says > Till was architected to follow best practices that DataHen has accumulated I know there are plenty o
by JulianWasTaken 5y ago
And only a few paragraphs down from a heading that says
> Till was architected to follow best practices that DataHen has accumulated
I know there are plenty of sites out there out to prevent any scraping whatsoever, or to improperly prevent some situations most of us would agree is reasonable behavior, but this appears to be blatantly hostile to web admins out there.
- cube00 5y agoI agree there are cases where entities don't want things scraped that really should be scraped to preserve the history they'd like to rewrite. My concern is that tools like these that use circumvention by default become the go-to when someone needs scraping and makes life hell for us running sites on hardware that's enough to service our small customer base but not an army of bots.
- paco3346 5y agoAdmin of a 5000+ website platform (with customer inventory) here. This the exact kind of thing I've been working so hard to block lately. Since Feb 2021 we've seen a substantial increase in scraping of our customers' inventory to the point we now have more bot traffic than human traffic. Annoyingly, only 5-10 requests come from a single IP so we've had to resort to always challenging requests that come from certain ASNs (typically those owned by datacenters). This type of project frustrates me because it knowlingly goes against a site's desired bot action (via robots.txt).