4 ms·

This is great, congrats on the savings. Do you have any opinions on the ethics of web scraping? I have some projects I want to pursue that involve web scraping

This is great, congrats on the savings. Do you have any opinions on the ethics of web scraping? I have some projects I want to pursue that involve web scraping but I'm worried about 1) any illegal technicalities 2) is it a dick move to the website owner

ta17711771 6y ago
> is it a dick move to the website owner FB gets more user engagement out of this, which is their goal. No harm, no foul.
mmcwilliams 6y ago
Not OP, but I find that the ethical questions tend to not be about web scraping itself and more what you are using it to accomplish. If you want to scrape and rehost some data then your ethical violation is in the copying and rehosting, not that you automated the downloading of data. It would be the same if you copied all the data by hand and then rehosted it. Then, of course, the way you go about it is important, too. Sapping the resources and bandwidth of a small site by sending it thousands of requests a minute is very different from automatically checking a public index once an hour.
snird 6y ago
I was a CTO of a company that did scraping to gather data and sell the analysis. We did this for sellers on Amazon. In short about legality - it depends on the country. But most will define user content as owned by the user, not the platform, therefore legal to scrape and use, as the user intents for the data to be public. But sites will fight you on this. Amazon specifically give up a really good fight. Ofcourse, this talks about user content in an "open" platform. If the content is owned by a business that has no intentions for it to be open for everyone in every medium, its probably illegal. In regard to being a dick move, it depends. If someone posted an ad in facebook, he intended for it to be seen. He doesn't care how. I just tunnel it to myself in easier way for me to consume.
- yowlingcat 6y ago
  I think there was also case law about this recently[1] regarding LinkedIn, where if the content isn't behind a login wall, it can't legally be considered anything but public. The UGC aspect is an interesting angle I haven't thought of, but that's a fantastic point too. Would I be correct in saying that the same mechanism companies set up to evade responsibility for illegal UGC also sets it up to be legal to scrape? And if the user owns the material, what happens if you've already made money selling that data and they ask for their cut of the proceeds?
tasuki 6y ago
1) No idea... probably location-specific too. 2) I ask the owner first (perhaps they even want to send me the data directly). If they don't respond, I go ahead and scrape, spreading the requests over a long time so as not hit their website too hard.

ta17711771 6y ago

> is it a dick move to the website owner FB gets more user engagement out of this, which is their goal. No harm, no foul.

mmcwilliams 6y ago

Not OP, but I find that the ethical questions tend to not be about web scraping itself and more what you are using it to accomplish. If you want to scrape and rehost some data then your ethical violation is in the copying and rehosting, not that you automated the downloading of data. It would be the same if you copied all the data by hand and then rehosted it. Then, of course, the way you go about it is important, too. Sapping the resources and bandwidth of a small site by sending it thousands of requests a minute is very different from automatically checking a public index once an hour.

snird 6y ago

I was a CTO of a company that did scraping to gather data and sell the analysis. We did this for sellers on Amazon. In short about legality - it depends on the country. But most will define user content as owned by the user, not the platform, therefore legal to scrape and use, as the user intents for the data to be public. But sites will fight you on this. Amazon specifically give up a really good fight. Ofcourse, this talks about user content in an "open" platform. If the content is owned by a business that has no intentions for it to be open for everyone in every medium, its probably illegal. In regard to being a dick move, it depends. If someone posted an ad in facebook, he intended for it to be seen. He doesn't care how. I just tunnel it to myself in easier way for me to consume.

yowlingcat 6y ago
I think there was also case law about this recently[1] regarding LinkedIn, where if the content isn't behind a login wall, it can't legally be considered anything but public. The UGC aspect is an interesting angle I haven't thought of, but that's a fantastic point too. Would I be correct in saying that the same mechanism companies set up to evade responsibility for illegal UGC also sets it up to be legal to scrape? And if the user owns the material, what happens if you've already made money selling that data and they ask for their cut of the proceeds?

tasuki 6y ago

1) No idea... probably location-specific too. 2) I ask the owner first (perhaps they even want to send me the data directly). If they don't respond, I go ahead and scrape, spreading the requests over a long time so as not hit their website too hard.