3 ms·

I thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's sto

by jd20 8y ago

I thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's stopping them from sending scrapehub a C&D letter and forcing them to comply?

ChuckMcM 8y ago
Sure, if you can make a reasonable assumption it is them scraping you. As they point out in the article they invest in proxy networks to make their requests appear to come from a bunch of addresses that don't lead back to them. One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. That put us as a web site in a bind in that the proxy service that was running on an infected computer was violating our terms of service but the user might be completely unaware. If they were also a customer and we black holed their IP it would also cut off legitimate traffic. Since we didn't keep a logs that could identify these relations over time (privacy issues) we had to rely on other methods. We never got enough penetration into the search market to make this a huge concern however so the problem remained largely theoretical. We started a program of doing exponential banning where an IP would be banned and then an hour later unbanned, and if it resumed its bad behavior banned for 2 hours then 4 Etc. Once you get to 1024 hours it is pretty safe to assume they are lawfully evil as it were. These guys fake their user agent, mask their IP addresses, and generally work hard to defeat anti-bot measures. They know they are over the line, but the law has yet to catch up to them.
- jd20 8y ago
  My experience, when it comes to scraping airline websites, the airline's legal department usually doesn't wait to have proof that you were the one that actually scraped them. If you have their data on your website, they send you a C&D, and if they continue to find their data on your website, they will happily sue you. In other words, doesn't matter how you got the data, you must've broken the law if you got their data. I'm thinking of RyanAir suing Expedia, United vs wandr.me, Southwest suing SWMonkey.com, I'm sure there's countless others.
  - bryanrasmussen 8y ago
    But if you have Amazon turk workers 'scraping' the data is that illegal?
    - nl 8y ago
      It’s the reproduction of the data which they are suing over, not the method.
deleted 8y ago
[deleted]

by jd20 8y ago

ChuckMcM 8y ago

Sure, if you can make a reasonable assumption it is them scraping you. As they point out in the article they invest in proxy networks to make their requests appear to come from a bunch of addresses that don't lead back to them. One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. That put us as a web site in a bind in that the proxy service that was running on an infected computer was violating our terms of service but the user might be completely unaware. If they were also a customer and we black holed their IP it would also cut off legitimate traffic. Since we didn't keep a logs that could identify these relations over time (privacy issues) we had to rely on other methods. We never got enough penetration into the search market to make this a huge concern however so the problem remained largely theoretical. We started a program of doing exponential banning where an IP would be banned and then an hour later unbanned, and if it resumed its bad behavior banned for 2 hours then 4 Etc. Once you get to 1024 hours it is pretty safe to assume they are lawfully evil as it were. These guys fake their user agent, mask their IP addresses, and generally work hard to defeat anti-bot measures. They know they are over the line, but the law has yet to catch up to them.

jd20 8y ago
My experience, when it comes to scraping airline websites, the airline's legal department usually doesn't wait to have proof that you were the one that actually scraped them. If you have their data on your website, they send you a C&D, and if they continue to find their data on your website, they will happily sue you. In other words, doesn't matter how you got the data, you must've broken the law if you got their data. I'm thinking of RyanAir suing Expedia, United vs wandr.me, Southwest suing SWMonkey.com, I'm sure there's countless others.
- bryanrasmussen 8y ago
  But if you have Amazon turk workers 'scraping' the data is that illegal?
  - nl 8y ago
    It’s the reproduction of the data which they are suing over, not the method.

deleted 8y ago

[deleted]