6 ms·

With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news

With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.

bryanrasmussen 5y ago
this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
Ian_Kerins 5y ago
100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
- travisporter 5y ago
  Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
  - RobSm 5y ago
    If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people steal my data!'
    - slaymaker1907 5y ago
      I disagree because there is a difference between posting something publicly for humans and posting something publicly for bots/large scale analysis. I'm ok with my employer possibly being able to see whether I am looking for a new job or not on LinkedIn if that means they would need to have a human looking at my LinkedIn page. I am not ok with them training some ML algorithm to monitor my LinkedIn page to determine how likely I am to leave the company at all times. Another danger is when public but not easily accessible data is able to deanonymize datasets which is probably the norm rather than the exception for anonymized datasets. Sure there are technical measures to make it better, but at the end of the day I think a lot of privacy is about respecting social boundaries and not breaking these protection measures even if technically possible. Most of the time, these measures are really about keeping honest people honest and not about stopping dedicated attackers.
    - LamaOfRuin 5y ago
      I have quite conscientiously never posted most of that information publicly, and yet it is for sale.
Terry_Roll 5y ago
You dont even need to do that, go overt plain sight in yer face and call yourself a search engine!
- joe_91 5y ago
  Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...
  - Terry_Roll 5y ago
    That was a cheap way to read the FT.com!
    - Ian_Kerins 5y ago
      Haha, nice hack!
  - fleddr 5y ago
    No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "It's public" is a legal defense, not an ethical one. It's public for readers, not for scrapers. It's public within the original context of the website, which may include monetization. Photographing every page of a book and then reading it that way may be legally allowed, but it's still unethical. I have somebody in our neighborhood that instead of paying for private trash, takes tiny bags of his private trash to the park and dumps it into the public trash cans. Legal? Yes. Parasitic behavior? Also yes.
    - LunaSea 5y ago
      > No they don't, Google and Bing respect robots.txt. They don't.

bryanrasmussen 5y ago

this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.

Ian_Kerins 5y ago

100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.

travisporter 5y ago
Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
- RobSm 5y ago
  If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people steal my data!'
  - slaymaker1907 5y ago
    I disagree because there is a difference between posting something publicly for humans and posting something publicly for bots/large scale analysis. I'm ok with my employer possibly being able to see whether I am looking for a new job or not on LinkedIn if that means they would need to have a human looking at my LinkedIn page. I am not ok with them training some ML algorithm to monitor my LinkedIn page to determine how likely I am to leave the company at all times. Another danger is when public but not easily accessible data is able to deanonymize datasets which is probably the norm rather than the exception for anonymized datasets. Sure there are technical measures to make it better, but at the end of the day I think a lot of privacy is about respecting social boundaries and not breaking these protection measures even if technically possible. Most of the time, these measures are really about keeping honest people honest and not about stopping dedicated attackers.
  - LamaOfRuin 5y ago
    I have quite conscientiously never posted most of that information publicly, and yet it is for sale.

Terry_Roll 5y ago

You dont even need to do that, go overt plain sight in yer face and call yourself a search engine!

joe_91 5y ago
Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...
- Terry_Roll 5y ago
  That was a cheap way to read the FT.com!
  - Ian_Kerins 5y ago
    Haha, nice hack!
- fleddr 5y ago
  No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "It's public" is a legal defense, not an ethical one. It's public for readers, not for scrapers. It's public within the original context of the website, which may include monetization. Photographing every page of a book and then reading it that way may be legally allowed, but it's still unethical. I have somebody in our neighborhood that instead of paying for private trash, takes tiny bags of his private trash to the park and dumps it into the public trash cans. Legal? Yes. Parasitic behavior? Also yes.
  - LunaSea 5y ago
    > No they don't, Google and Bing respect robots.txt. They don't.