3 ms·
You probably want the web equivalent of malicious compliance - an algorithmically generated web-hole or similar. That way the bot author isn't entirely sure you
by pbhjpbhj 9y ago
You probably want the web equivalent of malicious compliance - an algorithmically generated web-hole or similar. That way the bot author isn't entirely sure you're on to them; it could be a bot or server error. Like send the right headers but garbage data that looks like it's compressed but isn't, or doubly compressed garbage, or trim pages at a different place (before anything interesting), or slow data transfers, or ...
- psandersen 9y agoJust randomize the content they're trying the scrape imho
- dx034 9y agoI saw that with a page and it works extremely well. Inserting wrong data based on rules (there it was on rate limiting rather than user agent) is extremely hard to detect. The scraper never knows if they see real or wrong data. On the other hand, this will also get wrong data in search engines.
- imtringued 9y agoI think you could get pretty far just by simply scrambling the CSS classes and ids.
- justrobert 9y agoThe main thing headless chrome saves is having to spawn and manage xvfb per instance. If more sites adopt serving fake data to headless chrome, people will just return to the old xvfb workflow for those sites, and use headless for everything else. Could even use a small set of xvfb scrapers to verify the results, and automate detection of false data.