4 ms·

In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers

In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text

dsacco 9y ago
To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
- ivanhoe 9y ago
  The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
ChrisSD 9y ago
In addition to dsacco's reply it should be mention that using images instead of text hurts accessibility and, depending on your industry and country, it may also run afoul of equal access laws.
ris 9y ago
And there goes your accessibility.
ivanhoe 9y ago
Depending on a page and data it can make life a bit harder for the spider, but all that can be taken care of. IMHO the best way to stop spiders is to control access to your pages. Require users to register and then login each time to see the data, and then have some pro-active monitoring & counter measures in place, monitoring the patterns per users, and per IPs, and across the whole system for an unexpected increase in activity or bot-like behavior (moving too fast or at unusually uniform speed, following links too sequentially, etc.). If it's not an option, the next best approach is to make it hard for spider to fetch the whole dataset. Don't let user just browse all the pages, instead force them to use search, it makes it much harder to cover everything (and it will not affect normal users too much). You can, for example, return just the 100 products at once, and ask user to refine the search if there's more products than that. Then you can create some monitoring system to watch over unusual search queries that look like dictionary attacks. I've actually built a system like this for one client and it worked fairly well (combined with a few more tricks they were already using). Of course, all of this can be tricked too, it's all a game of cat & mouse, trying constantly to outsmart the other side.

dsacco 9y ago

To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...

ivanhoe 9y ago
The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.

ChrisSD 9y ago

In addition to dsacco's reply it should be mention that using images instead of text hurts accessibility and, depending on your industry and country, it may also run afoul of equal access laws.

ris 9y ago

And there goes your accessibility.

ivanhoe 9y ago

Depending on a page and data it can make life a bit harder for the spider, but all that can be taken care of. IMHO the best way to stop spiders is to control access to your pages. Require users to register and then login each time to see the data, and then have some pro-active monitoring & counter measures in place, monitoring the patterns per users, and per IPs, and across the whole system for an unexpected increase in activity or bot-like behavior (moving too fast or at unusually uniform speed, following links too sequentially, etc.). If it's not an option, the next best approach is to make it hard for spider to fetch the whole dataset. Don't let user just browse all the pages, instead force them to use search, it makes it much harder to cover everything (and it will not affect normal users too much). You can, for example, return just the 100 products at once, and ask user to refine the search if there's more products than that. Then you can create some monitoring system to watch over unusual search queries that look like dictionary attacks. I've actually built a system like this for one client and it worked fairly well (combined with a few more tricks they were already using). Of course, all of this can be tricked too, it's all a game of cat & mouse, trying constantly to outsmart the other side.