3 ms·
A lot of the people commenting on these techniques being fallible are missing the point: the idea isn't to make scraping impossible (despite the misleading titl
by cletus 14y ago
A lot of the people commenting on these techniques being fallible are missing the point: the idea isn't to make scraping impossible (despite the misleading title), it's to make it hard(er).
A determined scraper will defeat these techniques but most scrapers aren't determined, sufficiently skilled or so inclined to spend the time.
I've been curious about a variation of the honeypot scheme using something like Varnish. If you catch a scraper with a honeypot, how easy would it be to give them a version of your site that is cached and doesn't update very often?
- ChuckMcM 14y agoC'mon cletus give us a bit of credit. Are you telling us that a company with the World's Smartest Engineers(tm) doesn't already do exactly this with their custom front end machines? :-) It's one of the more entertaining new hire classes. You are correct that perfection is not achievable and you don't even want to get so close that you get very many false positives. But honey pots are bandwidth, which for folks who pay for bandwidth as part of their infrastructure charge, its a burden they are loath to bear. Rather to simply toss the packets into the ether whence they came rather than bother waking up their EC-2 instance.
- robryan 14y agoThey sure love to use GWT with an indecipherable exchange format though, have tried to scrape a few things in adwords before. I'm sure it is possible but there was enough of a deterrent for me to not bother.