3 ms·

So playing devils advocate, isn't it supposed to prevent crawlers (etc) from reading (ie accessing) those web pages?

• JohnFen 4y ago
  I'm not sure how the law would be interpreted, but from my technical viewpoint, robots.txt is not an access control at all because it makes no attempt at controlling access. It's merely advisory, no different than if you put a readme.txt that said "I don't want xxx to come here". There are other mechanisms to do actual access control if that's what the site owner wants.
  • darreninthenet 4y ago
    Still playing DA, would a notice page saying "only authorised user past this page <NEXT>" constitute an access control, in the strictest sense?
    • JohnFen 4y ago
      Not in my view. In my view, an "access control" is a mechanism that prevents access to unauthorized people. An advisory page like that is not preventing access, it's merely asking people not to access. I think I remember that there have been court cases that support this interpretation of the DMCA, but I'm not sure. Now, if that notice page had a login form or only allowed certain IPs to proceed, that would be an access control.