4 ms·

Correct me if I’m wrong,but there’s no law saying that you can’t crawl a site if it has a robots.txt configured, it’s just a convention, and you wouldn’t be abl

Correct me if I’m wrong,but there’s no law saying that you can’t crawl a site if it has a robots.txt configured, it’s just a convention, and you wouldn’t be able to have any recourse telling a company they can’t pull your info if it’s on the public web.

ktbwrestler 4y ago
Your best path forward might be trying to use some sort of reverse proxy that denies their egress IPs, but you could just be playing whack a mole
JohnFen 4y ago
This is correct. However, Apache (and, I assume, all other webservers) does let you examine the IP address and user agent strings and make decisions about how to handle them. This is how I stopped crawlers from my sites. I used robots.txt, but backstopped that with detecting undesirable IP addresses and user agent strings. Those would get an error page instead of the actual page. It does mean you need to keep an eye on your logs to spot new crawlers coming around, though.
darreninthenet 4y ago
I'm curious as to why it's not considered to be a DMCA circumvention of a technology.
- JohnFen 4y ago
  IANAL, but here's the relevant part of the law: “No person shall circumvent a technological measure that effectively controls access to a work protected under this title.” I'm guessing that because robots.txt does not actually attempt to prevent access, it also doesn't "effectively control access" and so isn't covered by this. The clear intent of the law is to prevent actual cracking of access controls.
  - darreninthenet 4y ago
    So playing devils advocate, isn't it supposed to prevent crawlers (etc) from reading (ie accessing) those web pages?
    - JohnFen 4y ago
      I'm not sure how the law would be interpreted, but from my technical viewpoint, robots.txt is not an access control at all because it makes no attempt at controlling access. It's merely advisory, no different than if you put a readme.txt that said "I don't want xxx to come here". There are other mechanisms to do actual access control if that's what the site owner wants.
      - darreninthenet 4y ago
        Still playing DA, would a notice page saying "only authorised user past this page <NEXT>" constitute an access control, in the strictest sense?
        JohnFen 4y ago
        Not in my view. In my view, an "access control" is a mechanism that prevents access to unauthorized people. An advisory page like that is not preventing access, it's merely asking people not to access. I think I remember that there have been court cases that support this interpretation of the DMCA, but I'm not sure. Now, if that notice page had a login form or only allowed certain IPs to proceed, that would be an access control.

ktbwrestler 4y ago

Your best path forward might be trying to use some sort of reverse proxy that denies their egress IPs, but you could just be playing whack a mole

JohnFen 4y ago

This is correct. However, Apache (and, I assume, all other webservers) does let you examine the IP address and user agent strings and make decisions about how to handle them. This is how I stopped crawlers from my sites. I used robots.txt, but backstopped that with detecting undesirable IP addresses and user agent strings. Those would get an error page instead of the actual page. It does mean you need to keep an eye on your logs to spot new crawlers coming around, though.

darreninthenet 4y ago

I'm curious as to why it's not considered to be a DMCA circumvention of a technology.

JohnFen 4y ago
IANAL, but here's the relevant part of the law: “No person shall circumvent a technological measure that effectively controls access to a work protected under this title.” I'm guessing that because robots.txt does not actually attempt to prevent access, it also doesn't "effectively control access" and so isn't covered by this. The clear intent of the law is to prevent actual cracking of access controls.
- darreninthenet 4y ago
  So playing devils advocate, isn't it supposed to prevent crawlers (etc) from reading (ie accessing) those web pages?
  - JohnFen 4y ago
    I'm not sure how the law would be interpreted, but from my technical viewpoint, robots.txt is not an access control at all because it makes no attempt at controlling access. It's merely advisory, no different than if you put a readme.txt that said "I don't want xxx to come here". There are other mechanisms to do actual access control if that's what the site owner wants.
    - darreninthenet 4y ago
      Still playing DA, would a notice page saying "only authorised user past this page <NEXT>" constitute an access control, in the strictest sense?
      - JohnFen 4y ago
        Not in my view. In my view, an "access control" is a mechanism that prevents access to unauthorized people. An advisory page like that is not preventing access, it's merely asking people not to access. I think I remember that there have been court cases that support this interpretation of the DMCA, but I'm not sure. Now, if that notice page had a login form or only allowed certain IPs to proceed, that would be an access control.