4 ms·

IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just

IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?

heipei 9y ago
There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/
- ant6n 9y ago
  What about tor.
  - heipei 9y ago
    TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
    - crtasm 9y ago
      I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
  - dsacco 9y ago
    Tor would introduce pretty intolerable latency into a professional data crawling setup.
- ficklepickle 9y ago
  I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...
tyingq 9y ago
>Having said that, just let them scrape? Depends on who you are. Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Letting them scrape with no limits is costly.
- dsacco 9y ago
  > Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Yes, and they make it very cumbersome to discourage it. I've successfully written scrapers for airlines. It's significantly more difficult than crawling other websites for a few reasons: 1. Session management is wacky - they really like to manage state entirely through cookies, and you typically need to visit a specific set of pages in a specific sequence before you can access the resources you want, like the number of seats available or their prices. 2. Sessions have time limits because anyone who looks at the seats initiates a "soft" reservation on them (this works in a similar way for theatre, concert and movie seating). 3. You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing (which, given the type of HTML you encounter, can be hell).
  - tyingq 9y ago
    >You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.
    - dsacco 9y ago
      Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
- IgorPartola 9y ago
  And why is that a good thing for them? I mean if they provided an API to get their pricing, wouldn't it be in their interest? Does anyone buy airline tickets from the airline website ever? I don't. I use sites like Expedia and Priceline. And if an airline is not listed or has shitty prices, I don't buy it. Wouldn't it be in their interest to be listed on there? Moreover, wouldn't they want the other companies to have similar API's to have dynamic pricing? I get that at one point not having this info out there was a good thing for them. But now the car is out of the bag. You can't keep pretending that booking sites don't exist.
jalfresi 9y ago
There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
jalfresi 9y ago
There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.

IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?

heipei 9y ago

There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/

ant6n 9y ago
What about tor.
- heipei 9y ago
  TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
  - crtasm 9y ago
    I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
- dsacco 9y ago
  Tor would introduce pretty intolerable latency into a professional data crawling setup.
ficklepickle 9y ago
I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...

tyingq 9y ago

>Having said that, just let them scrape? Depends on who you are. Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Letting them scrape with no limits is costly.

dsacco 9y ago
> Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Yes, and they make it very cumbersome to discourage it. I've successfully written scrapers for airlines. It's significantly more difficult than crawling other websites for a few reasons: 1. Session management is wacky - they really like to manage state entirely through cookies, and you typically need to visit a specific set of pages in a specific sequence before you can access the resources you want, like the number of seats available or their prices. 2. Sessions have time limits because anyone who looks at the seats initiates a "soft" reservation on them (this works in a similar way for theatre, concert and movie seating). 3. You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing (which, given the type of HTML you encounter, can be hell).
- tyingq 9y ago
  >You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.
  - dsacco 9y ago
    Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
IgorPartola 9y ago
And why is that a good thing for them? I mean if they provided an API to get their pricing, wouldn't it be in their interest? Does anyone buy airline tickets from the airline website ever? I don't. I use sites like Expedia and Priceline. And if an airline is not listed or has shitty prices, I don't buy it. Wouldn't it be in their interest to be listed on there? Moreover, wouldn't they want the other companies to have similar API's to have dynamic pricing? I get that at one point not having this info out there was a good thing for them. But now the car is out of the bag. You can't keep pretending that booking sites don't exist.

jalfresi 9y ago

There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.

deleted 9y ago
[deleted]
deleted 9y ago
[deleted]

jalfresi 9y ago

There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.