5 ms·

I worked on a site selling second-hand goods and there were around a dozen robots patrolling the site every day, completely ignoring the robots.txt (but fetchin

I worked on a site selling second-hand goods and there were around a dozen robots patrolling the site every day, completely ignoring the robots.txt (but fetching it in order to specifically look at disallowed links first). They homed in on any trademarks and kept fetching the pages repeatedly. Sounds like exceeding authorized access to me. That's grounds for jailarity, nowadays.

jalfresi 11y ago
robots.txt files are not mechanisms for authorizing access; they are guidelines designed to instruct user-agents on relevant resources. They essentially say to the user-agent "the content marked as 'disallow' should not be crawled for indexing purposes for unspecified reasons" Adhering to a robots.txt file is not mandatory, though it does make you a good netizen/curries favour with server admins.
- smitherfield 11y ago
  People have actually gotten into trouble for accessing subdirectories which aren't accessible via links from the site's index page. But I'm not sure if that extends to robots.txt, at least as it pertains to stuff which is intended for public consumption. Otherwise people who host sites containing illegal content would just put the entire site in robots.txt.
smitherfield 11y ago
So is selling counterfeit goods, so I'd guess they figure it's mutually-assured-destruction. And of course it's a lot easier to prove that someone is selling counterfeit goods from your brand than for them to prove that you intentionally wrote a web crawler to exceed unauthorized access. And IIRC it's considered legal to exceed unauthorized access in order to report a crime (I think that's been the case a few times with people who've hacked into computers and then reported child porn they've found to the Feds, which is kinda moral hazard IMO because it'd be very easy for a hacker to frame someone that way).

jalfresi 11y ago

robots.txt files are not mechanisms for authorizing access; they are guidelines designed to instruct user-agents on relevant resources. They essentially say to the user-agent "the content marked as 'disallow' should not be crawled for indexing purposes for unspecified reasons" Adhering to a robots.txt file is not mandatory, though it does make you a good netizen/curries favour with server admins.

smitherfield 11y ago
People have actually gotten into trouble for accessing subdirectories which aren't accessible via links from the site's index page. But I'm not sure if that extends to robots.txt, at least as it pertains to stuff which is intended for public consumption. Otherwise people who host sites containing illegal content would just put the entire site in robots.txt.

smitherfield 11y ago

So is selling counterfeit goods, so I'd guess they figure it's mutually-assured-destruction. And of course it's a lot easier to prove that someone is selling counterfeit goods from your brand than for them to prove that you intentionally wrote a web crawler to exceed unauthorized access. And IIRC it's considered legal to exceed unauthorized access in order to report a crime (I think that's been the case a few times with people who've hacked into computers and then reported child porn they've found to the Feds, which is kinda moral hazard IMO because it'd be very easy for a hacker to frame someone that way).