6 ms·

You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behin

You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.

manigandham 11y ago
What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
- fauigerzigerk 11y ago
  >What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
  - manigandham 11y ago
    Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
    - fauigerzigerk 11y ago
      ">Putting a up a webserver that can be publically reached is not authorization to access it." This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server. ">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to" You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page. ">We already have an explicit signal called robots.txt" Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver. [Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.
      - manigandham 11y ago
        1) You're not a lawyer. https://news.ycombinator.com/item?id=10339328 https://news.ycombinator.com/item?id=10339328 2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins." 3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id=14614 http://www.net-security.org/secworld.php?id=14614 I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.
        fauigerzigerk 11y ago
        >You're not a lawyer And you are a lawyer? >This directly contradicts your previous comment Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways. >There is already case law precedent regarding this exact type of publically accessible information not being authorized You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose. I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.

manigandham 11y ago

What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.

fauigerzigerk 11y ago
>What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins. So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
- manigandham 11y ago
  Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
  - fauigerzigerk 11y ago
    ">Putting a up a webserver that can be publically reached is not authorization to access it." This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server. ">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to" You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page. ">We already have an explicit signal called robots.txt" Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver. [Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.
    - manigandham 11y ago
      1) You're not a lawyer. https://news.ycombinator.com/item?id=10339328 https://news.ycombinator.com/item?id=10339328 2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins." 3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id=14614 http://www.net-security.org/secworld.php?id=14614 I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.
      - fauigerzigerk 11y ago
        >You're not a lawyer And you are a lawyer? >This directly contradicts your previous comment Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways. >There is already case law precedent regarding this exact type of publically accessible information not being authorized You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose. I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.