10 ms·
Yelp Claims Google Broke Promise Made to Regulators
- latchkey 9y agohttps://yelp.com/robots.txt https://yelp.com/robots.txt There is some fun stuff in there, like this?! Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0 Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1
- graphememes 9y agoThey assume the human who created the robot is built to Asimov's law.
- toomuchtodo 9y agoSupposedly a honeypot: https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338 Also appears they block archive.is outright (which ignores robots.txt).
- yeukhon 9y agoCan anyone explain how that honeypot works? i.e. how would anyone try to go to this page?
- gkoberger 9y agorobots.txt is a publicly accessible list of pages that bots aren't allowed to go to. It's like publishing a list of home addresses you don't want robbed because you have something good there. So, some bad crawlers might go to the robots.txt since it tells you where the good stuff is hidden. (In reality, it's likely nobody would put actual secret URLs in the robots.txt, but some developers might without thinking. This is likely just one of many ways Yelp attempts to catch bad webcrawlers.)
- ryanbrunner 9y agoI think it's more likely that they have hidden links and references to these URLs scattered across their "normal" pages. A human would never click on them, but any bot that's just blindly following links would stumble across them, which alerts Yelp that it's likely a bot accessing the site.
- ageitgey 9y agoTo catch and block scripts scraping Yelp in ways they don't like. If you fetch that url, you are most likely a bot that isn't following the rules and can be flagged or blocked outright without impacting any real users or search engine rankings no matter what your user agent claims to be.
- yeukhon 9y agoI see. Thanks, but then why so many?
- timr 9y agoTo make it hard for people to write code to get around it.
- yeukhon 9y agoWell, wouldn't that only be very effective if Yelp update this file every so often? But this line was added as early as 1425 days ago (https://news.ycombinator.com/item?id=6573338 https://news.ycombinator.com/item?id=6573338).
- timr 9y agoI assume that it's supposed to be regularly updated, yes.
- raverbashing 9y agoI wonder if any of their pages has a hidden link to one of those entries in robots.txt
- deleted 9y ago[deleted]
- striking 9y agoArchive.is is designed that way, though. It performs requests on behalf of a user, acting as a proxy and setting X-Forwarded-For with the user's IP address.
- thanksgiving 9y agoJust tried to archive the robots txt without even thinking lest it changes but can't. It is stuck. Screw Yelp. Good riddance. I hope they go bankrupt. It is clear this is someone (yes I'll say it: M$) using Yelp as a front to attack Google. So much for the new Microsoft. I'm ashamed to be a .net developer.
- ikeboy 9y agoSo I can load that url on a high-traffic page and IP block all visitors to my page from yelp?
- schwabacher 9y agoBe careful w/ these urls - I visited one and it appears my IP is now blocked from accessing yelp.com.
- cft 9y agoI just visited all of them: the IP was not blacklisted
- executesorder66 9y agoWorked on my laptop, but when I tried the same thing on my phone, I was not blocked. (even when I request the desktop version of the site)
- sitkack 9y agoBecause they whitelist the exit IP for the phone network, else one person could block access for millions.
- 013a 9y agoThat's really interesting. They might be trying to blacklist scrapers that don't properly respect robots.txt files.
- a012 9y agoI assume the GP was using their browser to access these links, though they still be blacklisted by Yelp?
- wickawic 9y agoScrapers can be pretty shady, so they're isn't a good way to ensure that web traffic is coming from a legitimate human using a browser. To the server it's all just bits on the wire.
- tossaway1 9y agoWhy would a scraper who doesn't respect robots.txt be accessing that file?
- Avitas 9y agoI am guessing that this is going to end up be a PR nightmare for Yelp once Google let's the hounds loose. They have been such an easy target in the past.
- ballenf 9y agoDon't know if Yelp does allow any search engines to scrape, but if they'd allow duckduckgo to scrape and help build some marginal competition against Google, it might give them more leverage overall.
- cabalamat 9y agoLooking at their robots.txt, they allow Google but not DDG.
- Ajedi32 9y agoHuh... Looks like they have a whitelist that allows Bing, Google, and Yandex (among others), but not DuckDuckGo: https://yelp.com/robots.txt https://yelp.com/robots.txt Does DuckDuckGo respect `robots.txt` files with onerous implicit restrictions like that? If I were them, I'd probably just write it so that if Googlebot is allowed then DuckDuckBot is too.
- thefreeman 9y agoSince when is Yelp a Google rival?
- godzillabrennus 9y agoGoogle bought Zagat
- pirocks 9y agoSince Google+ reviews show up in search results for things like restaurants.
- strict9 9y agoUsed to like Yelp as an alternative to Google owning everything, but they are destroying their brand using shady app-hijacking tactics. It's a dark pattern to limit access/force app downloads for access to content on mobile. I hate rooting for google, but I'll root for open web any day.
- graphememes 9y agoHeh, you should hear their shady business tactics. It would be a shame if your business suddenly had 1 star on yelp.
- krisdol 9y agoI'm unconvinced. I have read a ton of anecdotes and 2nd-hand accounts about this, but I've never seen any experiment or any piece of hard proof that yelp or its representatives hold your rating over your head if you don't sign up for advertising. It seems like an easy scenario to test out. I'm totally open to being convinced, but I'm skeptical right now that they are in fact that shady. I'm more inclined to believe that it's a placebo effect from suspicious people suddenly paying more attention to ratings after a sales call, and also attributing any negative dip in rating to the phone call they recently had. There are also third-party groups who call claiming to be able to list your business higher or rate it better, and I'm suspicious that some people confound these callers as yelp representatives. And linked from elsewhere in the thread: http://www.latimes.com/business/la-fi-yelp-ratings-20140905-story.html http://www.latimes.com/business/la-fi-yelp-ratings-20140905-... -- it just seems like with no scientific study backing the claims and case after case being lost, I think the onus of proof is on the people accusing yelp of these threats.
- teen 9y agoI've seen it first hand. My friend is a veternarian who was working in a very expensive neighborhood in California. Every time he got a 5 star review, it wouldn't display on his Yelp page. A few 1 star reviews did appear though. Yelp would call him and tell him essentially the 5 stars would reappear if he advertised with them. He had no choice but to pay and they all appeared, driving his average back up. How would you propose to prove it? And what would proving accomplish since many folks already believe it?
- henvic 9y agoRemoving Yelp from my phone. Never liked it anyways. Love Google and Foursquare.
- option_greek 9y agoYelp and Whiny seem to really rhyme together.
- spraak 9y agoYelp is a Rival to Google...? This claim surprises me.
- chaosite 9y agoGoogle is so big it's basically a rival to any company that sells computer software or services.
- fortythirteen 9y agoThe only rival for business reviews, which is almost exclusively a sub-domain of map apps now. Yelp has the reviews in Apple maps.
- face_mcgace 9y agoHmm these comments really bother me - people seem to miss the point that Google is using its monopoly power to block out competition. This concept is very, very frightening and we should all be worried. This is no different than when railroads would block competition and promote their own products.
- tossaway1 9y agoDoesn't Yelp have a reputation for much shadier business practices (i.e. extortion)? Maybe they're not a very sympathetic underdog in this fight.
- ocdtrekkie 9y agoYelp is definitely not a sympathetic party, but that doesn't invalidate or weaken their claim. Particularly if, as they claim, Google is specifically violating an agreement with the FTC.
- sitkack 9y agoYelp is indeed sht company and needs to be investigated for its own shakedown techniques. But this doesn't mean that Google isn't behaving in an anti-competitive way. Side note, Yelp literally called my partner who ran PR for non profits art org and asked to be paid $$ or else competitors ads would run on their page. Literal words. One would pay for NOTHING except not having competitors getting promoted on your own pages. Partner laughed and hung up.
- face_mcgace 9y agoYea perhaps - I'm not really defending Yelp as much as I'm worried about Google. I think this extends beyond just Yelp into the wider market (hotels.com, tripadvisor, paid search results, etc).
- buckhx 9y agoI'm not a Yelp sympathizer, but this does pretty clearly look like a violation on Google's side. Building up local content is hard and expensive and to represent someone else's content as their own is pretty unethical. If Google wants to present Yelp's content, they need to come up with a deal or grow their own photography content the hard way.
- samfriedman 9y agoNon-paywall link (twitter redirect): https://t.co/5sCeMykOWW https://t.co/5sCeMykOWW
- bl4ng 9y agoDid you know the name Yelp comes from "Yellow Pages"?