5 ms·
Can anyone explain how that honeypot works? i.e. how would anyone try to go to this page?
by yeukhon 9y ago
Can anyone explain how that honeypot works? i.e. how would anyone try to go to this page?
- gkoberger 9y agorobots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden. (In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)
- ryanbrunner 9y agoI think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.
- ageitgey 9y agoTo catch and block scripts scraping Yelp in ways they don't like. If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.
- yeukhon 9y agoI see. Thanks, but then why so many?
- timr 9y agoTo make it hard for people to write code to get around it.
- yeukhon 9y agoWell, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338).
- timr 9y agoI assume that it's supposed to be regularly updated, yes.
- raverbashing 9y agoI wonder if any of their pages has a hidden link to one of those entries in robots.txt
- deleted 9y ago[deleted]