4 ms·
Quora Blocks Startup Search Engines
- jay_kyburz 16y agoQuestion: Are smaller search companies allowed to build off of the results of other search providers, or is this blocked by the TOS.
- deleted 16y ago[deleted]
- alanh 16y agoGoogle in particular absolutely does not allow automated search queries.
- jpwagner 16y agoI didn't know that, can you point to a ref doc for this? thanks!
- jws 16y agoThe Google TOS: http://www.google.com/accounts/TOS http://www.google.com/accounts/TOS 5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
- hinathan 16y agoHere ya go: http://lmgtfy.com/?q=google+tos+robots&l=1 http://lmgtfy.com/?q=google+tos+robots&l=1
- jpwagner 16y agosee the above comment from jws that cites the appropriate text. here ya go: http://lmgtfy.com/?q=how+to+be+helpful http://lmgtfy.com/?q=how+to+be+helpful
- dasil003 16y agoWhy does http://lmgtfy.com/ http://lmgtfy.com/ still exist then?
- unshift 16y agolmgtfy just issues a redirect, it doesn't perform a query and return results of its own
- benologist 16y agoYou don't get your results on their site, they redirect you to Google.
- aneesh 16y agoNot quite true. They do allow automated queries through their APIs for at least some of their search verticals. http://code.google.com/more/#google-search http://code.google.com/more/#google-search
- Samuel_Michon 16y agoYahoo has an API to let other companies do exactly that. DuckDuckGo uses it. http://developer.yahoo.com/search/boss/ http://developer.yahoo.com/search/boss/
- citricsquid 16y ago"...Google, Bing, Blekko and other big players access..." Wait, since when was Blekko big? Also, why has every article about search engines recently ended up talking about DDG? :p
- deleted 16y ago[deleted]
- hitonagashi 16y agohttp://www.paulgraham.com/submarine.html http://www.paulgraham.com/submarine.html
- lukeschlather 16y agoI would expect an article about search engines to mention all the credibly useful search engines. Though I usually use Google, if we remove that as an option my next choice is DDG.
- rudiger 16y agoBlekko have around 800 servers in their data center, each with 64 GB RAM and eight SATA drives. http://www.readwriteweb.com/hack/2010/12/the-secrets-behind-blekkos-search-technology.php http://www.readwriteweb.com/hack/2010/12/the-secrets-behind-... That's pretty big in my book.
- mayank 16y agoI think he meant in terms of traffic and popularity. If you're measuring hardware, "large" is so very relative. Someone's going to beat you with a bigger stick, unless you're the NSA.
- redthrowaway 16y ago>why has every article about search engines recently ended up talking about DDG? Because they're the new darling, and tech types like them. It's always nice to see someone come in as a substantial underdog and make a real effort to succeed. Also, their results seem to be at least as good as Google's which is pretty impressive.
- shirtless_coder 16y agoI don't blame them for doing this. They are trying to protect their assets (questions and answers). However, this is not a good move for two reasons: 1. Honoring robots.txt is optional, so this doesn't actually offer any form of asset protection. 2. They are losing serious amounts of exposure that they would reap if they'd lift this restriction so that smaller engines can find them.
- joe_the_user 16y agoThis is indeed a disturbing, oligopolistic trend. Craiglist policy of specifically disallowing only classified search engines should look kinder in retrospect. It is reasonable to block sites which aim to merely re-frame your content within a "search" parameter. But that kind of thing can be dealt with terms of service. The problem is that it takes time to actively filter against offenders.
- terryjsmith 16y agoSpeaking from experience, this is why robots.txt files don't work; they've become anti-competitive in nature. If you actually build code to obey them you get locked out of actual sites, while the real scrapers you don't want don't listen to robots.txt anyways. If a site really doesn't want you there, they can block you at the IP/User-Agent level, which is what Quora will end up doing.
- joblessjunkie 16y agoWhen our startup was just getting going, search spiders were an unexpected problem for us. Our project was a bit odd, in that we had millions of documents available to browse. We wanted them all to be available to search engines, but it was an operational headache to keep that much data moving. Our normal daily traffic might have been 10,000 users with perhaps 50K page requests. We built our first small web server to handle this load. We were caught blindsided by the Googlebot, which from day one demanded hundreds of thousands of pages per day. Googlebot offers no option to slow it down, and if you try to slow it down on purpose your search ranking will suffer. So 90% of our early operational burden was devoted to keeping Googlebot happy. This was an unexpected burden, but it was worth it to be in the Google index, because they sent a lot of traffic our way and kept our fledging startup alive. But it was definitely not worth it to spend equivalent operational expense to be in marginal search engines that don't refer much traffic. To a site operator, they don't offer any value in exchange for what they demand. It was very interesting to see the relative volume of requests that we got from Googlebot versus the other crawlers. While Googlebot found us on day one, it took over 3 months before we appeared in Microsoft's index, and even then, they had perhaps only 10% of the coverage of our site when compared to Google.
- ig1 16y agoYou can slow down Googlebot via the Google Webmaster Tools
- joblessjunkie 16y agoAt the time, I recall that our choices were between "too fast" or "way, way too fast".
- rorrr 16y agoYou can request to change the crawl rate. http://www.google.com/support/webmasters/bin/answer.py?answer=48620 http://www.google.com/support/webmasters/bin/answer.py?answe...
- palish 16y ago
- alphadog 16y agoWhat's the penalty (if any) for ignoring robots.txt and crawling anyway? Any citations of it being legally enforced via TOS or ASP?
- quag 16y agoThere is no penalty, but they can always block the crawler or serve rubbish.
- moultano 16y agorobots.txt in the past has been the legal defense against claims of copyright infringement.
- uxp 16y agoCurious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respect.
- moultano 16y agoIANAL and looking for other sources, but it's discussed along with many other factors here: http://www.benedict.com/Digital/Internet/Field/Field.aspx http://www.benedict.com/Digital/Internet/Field/Field.aspx Essentially, the existence of robots.txt and meta noindex has made courts more comfortable ruling that the index of search engines constitutes fair use.
- uxp 16y agoBut this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them. Interesting link, however. I was unaware of that case.
- Samuel_Michon 16y agoFrom the article: "[Quora's] robots.txt file explicitly grants Google, Bing, Blekko and other big players access" Blekko is a big player? It launched two months ago and it's still in public beta. According to Compete.com [1], it attracts only 120,000 visitors a month. I'd say that if Blekko is a big player, DuckDuckGo is one too. [1] http://siteanalytics.compete.com/blekko.com/?metric=uv http://siteanalytics.compete.com/blekko.com/?metric=uv
- rudiger 16y agoBlekko has around 800 servers, each with 64 GB RAM and 8 TB of storage across SATA drives. http://www.readwriteweb.com/hack/2010/12/the-secrets-behind-blekkos-search-technology.php http://www.readwriteweb.com/hack/2010/12/the-secrets-behind-... Has DuckDuckGo published the details of their data-center anywhere? Also, Blekko may have launched recently, but it was founded in mid-2007.
- bobf 16y agoI agree wholeheartedly. For comparison, I currently see ~95% of search engine referral traffic from Google, 2.8% from Yahoo, 1.4% from Bing, and <1% from all other search engines combined. It makes no sense to let X, Y, Z random search engines index content like Google, when they send nowhere near the traffic. As others have mentioned, blocking by robots.txt often doesn't work as some bots don't obey it. I currently use an extensive rewrite rule to block those bots' User-Agents, and block specific IPs as a last resort.
- pilif 16y agoProblem is, that by disallowing those smaller players, you are preventing them from getting bigger because they would need to be able to index sites to actually become good enough to attract visitors. Also, when disallowing a spider you can be sure that you won't get ANY visitors from them, as you are not in their index to start with. I mean: if I disallow any bot but Googlebot, I will have 100% Google referrals by design. I can understand that in an emergency situation you might want to block some bots, but blocking all of them because they are currently small feels a) a bit unfair due to not giving them the chance to grow and b) shortsighted due to never knowing if they might be getting big enough or the visitors they deliver might be better ad-clickers
- wslh 16y agoIs robots.txt against net neutrality? net neutrality for bots.
- jmount 16y agoJust about everybody effective blocks (or throttles to near zero) all but the top 3 or so crawlers. For a popular commerce site a significant fraction (like 1/3rd) of the infrastructure capacity is just responding to crawler traffic.
- mmaunder 16y agoRunning a scraping site pretending to be a search engine is an oldy, but a goody. http://www.google.com/#sclient=psy&hl=en&q=site:duckduckgo.com http://www.google.com/#sclient=psy&hl=en&q=site:duck... Way to play the openness card Gabriel.
- dedward 16y agoDid anyone look at the robots.txt? It looks to me like it doesn't block anything - it does have some specific settings tailored for various big-boy search engines, with a catch-all rule at the end for those not otherwise defined. It also says right there at the top "# If you operate a search engine and would like to crawl Quora, please # email info@quora.com. Thanks. " So they're trying to get the right data to the right search engines the right way - makes sense to me. User-agent: * Allow: /$ Allow: /about$ Allow: /about/ Allow: /jobs$ Allow: /challenges$ Allow: /press$ Allow: /login$ Allow: /login/ Allow: /signup$
- jarek 16y agoThe important part is Disallow: / right after the allows.
- niyazpk 16y agoDid anyone actually go through the robots.txt file? http://www.quora.com/robots.txt http://www.quora.com/robots.txt The first two lines are: # If you operate a search engine and would like to crawl Quora, please # email info@quora.com. Thanks. Looks sensible to me.
- joe_the_user 16y agoAh, but do you know what the results of that emailing have been? Five bots are listed. Perhaps they're the only one who bothered to email. Or perhaps not.
- dedward 16y agoWhat about the rule that matches "*" at the end? That's there for all the rest - the others are just fine-tuned for mutual benefit. There is no rule in there that denys other search engines, just some rules that set some slightly different rules for different ones. EDIT: I see now -therule at the end is rather specific, and wouldn't allow crawling the entire site. Still - without actual complaints from someone who contacted them and got no response, I see no evil.
- jdp23 16y agoDid you actually read the article? EDIT: my bad. I got confused between Facebook and Quora
- joe_the_user 16y agoCould you clarify what the line: Disallow: / means?
- michaelbuckbee 16y agoIt's indicating the root of the site. Documentation for further reading at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html
- dedward 16y agoSearch engines can be a real PITA for a site. They can take a site down. When it's my site - I don't have to let you, or anyone else, index it. We can do it nicely through robots.txt, or cold-war style through firewalls and script and tarpits and who knows what else. So - assuming Quora took the time to put what they did in their robots.txt, in a world where many sites still don't bother at all - one can assume they are paying close attention to the business value they are extracting from search engine driven traffic.
- rwhitman 16y agoAggressive spiders are a pain in the ass. I'm with them on this.
- mtogo 16y agoSearch engine spiders, too.
- arihant 16y agoIf they are blocking to save resources, then it might be worth noting that startup search engines will not bomb their servers the way Google and Bing can. Are there any published numbers as to how the crawling load varies from startups to Googlebot?