5 ms·

I do a lot of crawler services as well. The last one, I did 13 crawlers to keeping comparing prices of all drugstore products online. I'm selling it to drugsto

I do a lot of crawler services as well. The last one, I did 13 crawlers to keeping comparing prices of all drugstore products online. I'm selling it to drugstore ecommerces who wants to know when their competitor prices and when they're doing promotions, if competitors prices are higher... Well, I just automated the job of a person that was doing this job manually every single day, looking into competitor's site If it is illegal, you have to ask to google maps to remove all houses from your database and just let in the houses who gave the permission to it. The both sentence means the same bullshit, they are both wrong: Google Maps: "The front-door of my house is faced to a public street. That does not mean you can take photos of it on Street View and use a very smart OCR, to read my house number. Not to mention that sometimes you give my house photo to others by Captcha asking the house number, c'mon!" Linked: "My CV is half-public. That does not mean you can take crawler of it." ... This is only a cool discussion in North Korea or maybe in China. Not in the rest of the world.

groby_b 9y ago
I would suggest occasionally actually informing yourself of the wider world. Your very example of Google Maps in Germany - well, turns out Google had to give people the option to opt out: https://europe.googleblog.com/2010/10/how-many-german-households-have-opted.html https://europe.googleblog.com/2010/10/how-many-german-househ... Same goes for web sites - robots.txt exists for a reason. If your crawler ignores that, well, I'd suggest talking to a lawyer.
- briandear 9y ago
  I wasn’t aware that robots.txt had any legal meaning. Am I wrong?
  - jasonkostempski 9y ago
    If you are wrong then there is no sanity left in the world.
  - pdabbadabba 9y ago
    > Am I wrong? I hate to be the bearer of bad new but...maybe. A reading of the Computer Fraud and Abuse Act could make robots.txt legally enforceable. And given the government's approach to CFAA cases a very aggressive interpretation, under the right circumstances (for example, when it provides evidence that the scraper knew that scrapint was not authorized), seems like a real possibility. Among the many other things CFAA criminalizes, it makes it a crime to "intentionally access[] a computer without authorization or exceed[] authorized access . . and thereby obtain[] information from any protected computer;" A "protected computer" is, among other things, a computer "which is used in or affecting interstate or foreign commerce or communication." That would probably cover just about any server on the Internet. https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030
    - hackits 9y ago
      I wouldn't be too fast to jump to conclusion robots.txt legally enforceable. You would need to cite prior case law's. Without any case law's it make decision on a law error prone at best.
      - solarkraft 9y ago
        The possibility still exists. After reading this (insanely broad) definition I think the chance is not even that low.
    - pkilgore 9y ago
      Saying something is illegal under the CFAA isn't really a stretch, it's one of the broadest statutes ever written. If you have an employee handbook at your job right now and are using the internet for your own entertainment you may be violating the CFAA. The interesting question is are you violating the CFAA in a way that will cause the executive branch to exercise their discretion to prosecute, and moreso, is the CFAA even constitutional.
  - groby_b 9y ago
    I'd talk to a lawyer. There are presumably some legal arguments that might apply. There definitely have been cases revolving around it, but I have no idea what the outcomes were. From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing. And often, "not a nice thing" translates into legal action. So, if this matters to you, you should find an expert :)
    - pkilgore 9y ago
      > From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing. As a lawyer asked inane questions by friends, a common answer is: "I am no sure if its legal to do, but I'm absolutely sure you're a asshole if you do it."

groby_b 9y ago

I would suggest occasionally actually informing yourself of the wider world. Your very example of Google Maps in Germany - well, turns out Google had to give people the option to opt out: https://europe.googleblog.com/2010/10/how-many-german-households-have-opted.html https://europe.googleblog.com/2010/10/how-many-german-househ... Same goes for web sites - robots.txt exists for a reason. If your crawler ignores that, well, I'd suggest talking to a lawyer.

briandear 9y ago
I wasn’t aware that robots.txt had any legal meaning. Am I wrong?
- jasonkostempski 9y ago
  If you are wrong then there is no sanity left in the world.
- pdabbadabba 9y ago
  > Am I wrong? I hate to be the bearer of bad new but...maybe. A reading of the Computer Fraud and Abuse Act could make robots.txt legally enforceable. And given the government's approach to CFAA cases a very aggressive interpretation, under the right circumstances (for example, when it provides evidence that the scraper knew that scrapint was not authorized), seems like a real possibility. Among the many other things CFAA criminalizes, it makes it a crime to "intentionally access[] a computer without authorization or exceed[] authorized access . . and thereby obtain[] information from any protected computer;" A "protected computer" is, among other things, a computer "which is used in or affecting interstate or foreign commerce or communication." That would probably cover just about any server on the Internet. https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030
  - hackits 9y ago
    I wouldn't be too fast to jump to conclusion robots.txt legally enforceable. You would need to cite prior case law's. Without any case law's it make decision on a law error prone at best.
    - solarkraft 9y ago
      The possibility still exists. After reading this (insanely broad) definition I think the chance is not even that low.
  - pkilgore 9y ago
    Saying something is illegal under the CFAA isn't really a stretch, it's one of the broadest statutes ever written. If you have an employee handbook at your job right now and are using the internet for your own entertainment you may be violating the CFAA. The interesting question is are you violating the CFAA in a way that will cause the executive branch to exercise their discretion to prosecute, and moreso, is the CFAA even constitutional.
- groby_b 9y ago
  I'd talk to a lawyer. There are presumably some legal arguments that might apply. There definitely have been cases revolving around it, but I have no idea what the outcomes were. From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing. And often, "not a nice thing" translates into legal action. So, if this matters to you, you should find an expert :)
  - pkilgore 9y ago
    > From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing. As a lawyer asked inane questions by friends, a common answer is: "I am no sure if its legal to do, but I'm absolutely sure you're a asshole if you do it."