5 ms·
https://yelp.com/robots.txt https://yelp.com/robots.txt There is some fun stuff in there, like this?! Disallow: /biz/outlook-autumn-market-fundamental
by latchkey 9y ago
https://yelp.com/robots.txt https://yelp.com/robots.txt
There is some fun stuff in there, like this?!
Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0
Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1
- graphememes 9y agoThey assume the human who created the robot is built to Asimov's law.
- toomuchtodo 9y agoSupposedly a honeypot: https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338 Also appears they block archive.is outright (which ignores robots.txt).
- yeukhon 9y agoCan anyone explain how that honeypot works? i.e. how would anyone try to go to this page?
- gkoberger 9y agorobots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden. (In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)
- ryanbrunner 9y agoI think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.
- ageitgey 9y agoTo catch and block scripts scraping Yelp in ways they don't like. If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.
- yeukhon 9y agoI see. Thanks, but then why so many?
- timr 9y agoTo make it hard for people to write code to get around it.
- yeukhon 9y agoWell, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338).
- timr 9y agoI assume that it's supposed to be regularly updated, yes.
- raverbashing 9y agoI wonder if any of their pages has a hidden link to one of those entries in robots.txt
- deleted 9y ago[deleted]
- striking 9y agoArchive.is is designed that way, though. It performs requests on behalf of a user, acting as a proxy and setting X-Forwarded-For with the user's IP address.
- thanksgiving 9y agoJust tried to archive the robots txt without even thinking lest it changes but can't. It is stuck. Screw Yelp. Good riddance. I hope they go bankrupt. It is clear this is someone (yes I'll say it: M$) using Yelp as a front to attack Google. So much for the new Microsoft. I'm ashamed to be a .net developer.
- ikeboy 9y agoSo I can load that url on a high-traffic page and IP block all visitors to my page from yelp?
- schwabacher 9y agoBe careful w/ these urls - I visited one and it appears my IP is now blocked from accessing yelp.com.
- cft 9y agoI just visited all of them: the IP was not blacklisted
- executesorder66 9y agoWorked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)
- sitkack 9y agoBecause they whitelist the exit IP for the phone network, else one person could block access for millions.
- 013a 9y agoThat's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.
- a012 9y agoI assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?
- wickawic 9y agoScrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.
- tossaway1 9y agoWhy would a scraper who doesn't respect robots.txt be accessing that file?