5 ms·

Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor ch

Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.

jordanpg 9y ago
In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text
- dsacco 9y ago
  To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
  - ivanhoe 9y ago
    The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
- ChrisSD 9y ago
  In addition to dsacco's reply it should be mention that using images instead of text hurts accessibility and, depending on your industry and country, it may also run afoul of equal access laws.
- ris 9y ago
  And there goes your accessibility.
- ivanhoe 9y ago
  Depending on a page and data it can make life a bit harder for the spider, but all that can be taken care of. IMHO the best way to stop spiders is to control access to your pages. Require users to register and then login each time to see the data, and then have some pro-active monitoring & counter measures in place, monitoring the patterns per users, and per IPs, and across the whole system for an unexpected increase in activity or bot-like behavior (moving too fast or at unusually uniform speed, following links too sequentially, etc.). If it's not an option, the next best approach is to make it hard for spider to fetch the whole dataset. Don't let user just browse all the pages, instead force them to use search, it makes it much harder to cover everything (and it will not affect normal users too much). You can, for example, return just the 100 products at once, and ask user to refine the search if there's more products than that. Then you can create some monitoring system to watch over unusual search queries that look like dictionary attacks. I've actually built a system like this for one client and it worked fairly well (combined with a few more tricks they were already using). Of course, all of this can be tricked too, it's all a game of cat & mouse, trying constantly to outsmart the other side.
hyder_m29 9y ago
I don't believe Google's Recaptcha2 has been broken yet (at least not publicly). Though you can parcel it off to low wage workers who click on captchas all day long.
Dolores12 9y ago
Every spider process is a browser tab. People have hundreds tabs opened without any problem on ordinal computers. On a dedicated server with 128GB of ram you can run thousands of those and it will cost you 100$-200$/month.
- jasonjayr 9y ago
  True -- But the user is not interacting with all 100's of those tabs at the same time -- something an automated scraper would be. Firefox + Chrome both lower the priority of background tabs, and may be doing other tricks so the background tabs can stick around and be switched to quickly.
  - Dolores12 9y ago
    Well, most of the time spiders are idle while waiting for IO operations to complete. Main bottleneck is not CPU, but RAM and bandwidth.
    - rileymat2 9y ago
      It depends on what analysis is being done on each page.
      - Dolores12 9y ago
        What kind of analysis are you expecting from a crawler which main purpose is to grab a webpage?
        ivanhoe 9y ago
        Main purpose is to render the page, crawl the dom for the data, and then load the next page as fast as possible... so it's equivalent of having a thousand bookmarks and opening them all at the same time, and as soon each loads running the scraper script and reloading them with the new link. Try it for yourself, it's easy to script, and check the load...
        9y ago
dsacco 9y ago
> If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. This is not true. I've seen it done in AWS for less than $2,000 monthly, and I can do it from my home for less than $500 per month (minus the costs of my workstation and networking gear, which you'd amortize over the expected lifetime of the project). I have a home server with 128GB of RAM and an i7-6900K, with 125 static IPs and a bunch of Ubiquiti networking gear. You don't even need that much memory or compute power, but I also use my workstation for other research projects. I use my own static IPs to parallelize without having to sacrifice latency or cede control to a shady proxy farm. It's a pretty straightforward setup - each static IP is given its own route across the switches from the gateway, and the switches have link aggregated connections for 40GbE bandwidth. You have pub sub and queuing, and each scraping target is its own headless Chrome process. The requests to the target from each process are round robin sent across the available interfaces. Then you've got parsing and a local database. I'm not a particularly invasive scraper (I make my User Agent deliberately obvious with an explicit way to opt out), but it's really not true that it's prohibitively expensive. This is a pretty cheap setup; I don't even profit from the work, I use it for personal research. If someone was actively selling valuable data, this would absolutely be worth it.
- toomuchtodo 9y ago
  Would you be willing to chat more about this one on one for research (non-commercial) purposes?
  - dsacco 9y ago
    Sure. I don't sell data but I'm happy to talk shop or help with interesting research projects.
- ivanhoe 9y ago
  It's a matter of a personal perspective. If you're based in US and have a luxury of working with enterprise clients than sure. On the other hand, few years ago when I was doing data scrapping as a freelancer, price tag of $2K just for the infrastructure would be a huge show stopper for the most of my average clients back then. We were charging them way less than that.

jordanpg 9y ago

In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text

dsacco 9y ago
To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
- ivanhoe 9y ago
  The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
ChrisSD 9y ago
In addition to dsacco's reply it should be mention that using images instead of text hurts accessibility and, depending on your industry and country, it may also run afoul of equal access laws.
ris 9y ago
And there goes your accessibility.
ivanhoe 9y ago
Depending on a page and data it can make life a bit harder for the spider, but all that can be taken care of. IMHO the best way to stop spiders is to control access to your pages. Require users to register and then login each time to see the data, and then have some pro-active monitoring & counter measures in place, monitoring the patterns per users, and per IPs, and across the whole system for an unexpected increase in activity or bot-like behavior (moving too fast or at unusually uniform speed, following links too sequentially, etc.). If it's not an option, the next best approach is to make it hard for spider to fetch the whole dataset. Don't let user just browse all the pages, instead force them to use search, it makes it much harder to cover everything (and it will not affect normal users too much). You can, for example, return just the 100 products at once, and ask user to refine the search if there's more products than that. Then you can create some monitoring system to watch over unusual search queries that look like dictionary attacks. I've actually built a system like this for one client and it worked fairly well (combined with a few more tricks they were already using). Of course, all of this can be tricked too, it's all a game of cat & mouse, trying constantly to outsmart the other side.

hyder_m29 9y ago

I don't believe Google's Recaptcha2 has been broken yet (at least not publicly). Though you can parcel it off to low wage workers who click on captchas all day long.

Dolores12 9y ago

Every spider process is a browser tab. People have hundreds tabs opened without any problem on ordinal computers. On a dedicated server with 128GB of ram you can run thousands of those and it will cost you 100$-200$/month.

jasonjayr 9y ago
True -- But the user is not interacting with all 100's of those tabs at the same time -- something an automated scraper would be. Firefox + Chrome both lower the priority of background tabs, and may be doing other tricks so the background tabs can stick around and be switched to quickly.
- Dolores12 9y ago
  Well, most of the time spiders are idle while waiting for IO operations to complete. Main bottleneck is not CPU, but RAM and bandwidth.
  - rileymat2 9y ago
    It depends on what analysis is being done on each page.
    - Dolores12 9y ago
      What kind of analysis are you expecting from a crawler which main purpose is to grab a webpage?
      - ivanhoe 9y ago
        Main purpose is to render the page, crawl the dom for the data, and then load the next page as fast as possible... so it's equivalent of having a thousand bookmarks and opening them all at the same time, and as soon each loads running the scraper script and reloading them with the new link. Try it for yourself, it's easy to script, and check the load...
      - 9y ago

dsacco 9y ago

> If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. This is not true. I've seen it done in AWS for less than $2,000 monthly, and I can do it from my home for less than $500 per month (minus the costs of my workstation and networking gear, which you'd amortize over the expected lifetime of the project). I have a home server with 128GB of RAM and an i7-6900K, with 125 static IPs and a bunch of Ubiquiti networking gear. You don't even need that much memory or compute power, but I also use my workstation for other research projects. I use my own static IPs to parallelize without having to sacrifice latency or cede control to a shady proxy farm. It's a pretty straightforward setup - each static IP is given its own route across the switches from the gateway, and the switches have link aggregated connections for 40GbE bandwidth. You have pub sub and queuing, and each scraping target is its own headless Chrome process. The requests to the target from each process are round robin sent across the available interfaces. Then you've got parsing and a local database. I'm not a particularly invasive scraper (I make my User Agent deliberately obvious with an explicit way to opt out), but it's really not true that it's prohibitively expensive. This is a pretty cheap setup; I don't even profit from the work, I use it for personal research. If someone was actively selling valuable data, this would absolutely be worth it.

toomuchtodo 9y ago
Would you be willing to chat more about this one on one for research (non-commercial) purposes?
- dsacco 9y ago
  Sure. I don't sell data but I'm happy to talk shop or help with interesting research projects.
ivanhoe 9y ago
It's a matter of a personal perspective. If you're based in US and have a luxury of working with enterprise clients than sure. On the other hand, few years ago when I was doing data scrapping as a freelancer, price tag of $2K just for the infrastructure would be a huge show stopper for the most of my average clients back then. We were charging them way less than that.