8 ms·

As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!

bduerst 11y ago
You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.
- jumperjake 11y ago
  IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them. I agree with the GP in that public content is fair game. How do you thing google works?
  - bduerst 11y ago
    Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags. Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.
manigandham 11y ago
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
- angry-hacker 11y ago
  You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.
  - manigandham 11y ago
    What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
    - fauigerzigerk 11y ago
      >What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
      - manigandham 11y ago
        Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.

bduerst 11y ago

You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.

jumperjake 11y ago
IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them. I agree with the GP in that public content is fair game. How do you thing google works?
- bduerst 11y ago
  Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags. Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.

manigandham 11y ago

That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.

angry-hacker 11y ago
You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.
- manigandham 11y ago
  What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
  - fauigerzigerk 11y ago
    >What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
    - manigandham 11y ago
      Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.