3 ms·
robots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because
by gkoberger 9y ago
robots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden.
(In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)
- ryanbrunner 9y agoI think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.