10 ms·
Google indexes Stack Overflow at a rate of 10 requests per second
- petervandijck 16y agoBecause that's what they asked Google to do in their Webmaster tools.
- cmontgomeryb 16y agoThat screenshot was to point out that when changing from "automatic" to "custom", there was no difference ie the setting Google's automatic setting settled on was "full pelt".
- androidlove 16y agoso what? so does every other seo optimizes page. (but how SO dealt with it was interresting nontheless)
- sajidnizami 16y agoSO is the best case study of a startup scaling with .Net. Whenever I read their infrastructure stuff I cringe I am not in their team.
- mey 16y agoThese articles may be insightful to you http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html http://highscalability.com/blog/2009/8/5/stack-overflow-arch... http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html http://highscalability.com/blog/2011/3/3/stack-overflow-arch... My two take aways are, generally you can make anything scale, cache like hell, and I personally don't see enough value in .NET to justify the licensing costs to roll it out initially or the long term.
- sajidnizami 16y agoThey are part of Bizspark program so they got free licenses. They don't even have much hardware. Important part isn't caching. You can just create HTML copies off a page and serve them through a fast server, that is a very easy cache. Difficult part is smart caching, cache getting updated when they need to be and persisted properly to the DB. These guys have nailed it in my opinion. I often lookup these guys and optimize my own sites and have been following them since their beta phase.
- jswinghammer 16y agoOne point on the cost issue. To an individual a Windows server license looks like a lot of money but to a business it really doesn't matter. Every server my company buys has many paying customers tied to it so it really doesn't matter to us at all.
- mey 16y agoThe cost of the Windows OS itself it not the real cost. SQLServer gets costly as you grow out, but not nearly as bad as Oracle. http://www.microsoft.com/sqlserver/2008/en/us/pricing.aspx http://www.microsoft.com/sqlserver/2008/en/us/pricing.aspx Yes, BizSpark makes it free, but that basically locks you into the platform long enough for you to rely on it before you start paying those prices.
- mseebach 16y agoBesides the monetary cost, there's the opportunity cost of dealing with licenses in the first place. Part is compliance (Does your company have current licenses that cover every bit of software on every virtual machine on every developer laptop? Can you prove it?) and part is procurement (Do you get the plan with free upgrades, or do you buy new? Will you need enough licenses over the next two years that you should get a site-license or is it cheaper to stick with single user licenses?).
- jdunck 16y agoCorrection: it doesn't matter for some workloads. This old classic applies to business models as well as software: http://www.joelonsoftware.com/articles/FiveWorlds.html http://www.joelonsoftware.com/articles/FiveWorlds.html
- simonw 16y agoIt's also a great case study of a hybrid model, where .Net application servers are complemented with open source components such as Redis and HAProxy.
- sajidnizami 16y agoExactly. Initially they weren't using anything OSS but moved to HAProxy and Redis. They even help update HAProxy from time to time. http://nosql.mypopescu.com/post/2669915777/powered-by-redis-stackoverflow-caches http://nosql.mypopescu.com/post/2669915777/powered-by-redis-...
- kristianp 16y agoNo mention of the "L" word in this thread. Stack(Overflow|Exchange) is running Linux! (and windows of course).
- sams99 16y agoScaling problems are not unique to .Net, you have them with Rails, PHP and even C++. As long as you are able to measure where your bottlenecks are and address them you are fine. I had a ball working on these performance issues and tuning down render times for question/show to 50ms (a totally dynamic page). I guess not all developers get a kick out of the same things.
- latch 16y agoI always thought that, at scale, SO and (Facebook or Twitter) were an apple to oranges comparison. Not because of load, but because of types of load. For whatever reason, I have it in my head that the difficulty Facebook and Twitter (and even Digg) face in scaling are the social aspects of their sites. These are the things that require custom software (FlockDB and Cassandra) and a lot of machines. Perhaps I need to use SO again, but in the day, this social aspect of SO didn't exist. This means their scaling challenges are far more traditional, say like slashdot. 99% cacheable reads type thing. If I'm right, SO is really just a case study that, depending on what they are doing, some startups will be able to scale with .NET.
- vain 16y agothis is a problem we had at a large social network i used to work with. launching a directory of users primarily for google's consumption was something that was difficult to scale for our huge size of database. The solution for us was node.js
- vain 16y agothis is a problem I had to solve at a large social network i used to work with. Launching a directory of users primarily for google's consumption was something that was difficult to scale for our huge size of database. The working solution I implemented was using node.js and pre-sorted data in mysql
- 3pt14159 16y agoWhy doesn't Stack Overflow make a push API that feeds Google changes to pages when A. they know they happen and B. when they arn't under peak load? (edit: besides the obvious limitations that Google may not have an API that is exposed to the public)
- simonw 16y agoI imagine the ONLY reason is that Google don't have such an API that is exposed to the public.
- pak 16y agoThere are services in Webmaster Tools for pushing sitemaps, no? Which would not be quite sufficient for this--Google still has to scrape the respective pages in the sitemap--but it's about as close as you are going to get.
- vladd 16y agoThere is an API via which you can describe your URLs (it's called a sitemap) and you can ping to Google your sitemap when its content changes. You can have multiple sitemaps and ping only your changes. More on www.sitemaps.org . But Google reserves the right to crawl non-sitemaps URLs, for obvious reasons. It would be quite a bad decision for them to restrict their crawls only to API-provided URLs.
- zach 16y agoWe try to give search engines hints with the update frequency in our sitemaps. We re-build our sitemaps nightly and make sure that new or recently-updated content is listed with an update frequency of "daily" or "weekly" and all other content pages are listed as being updated "monthly." To be honest, I've never measured if it works, but it can't hurt.
- eli 16y agoHow do you ping only changes? Don't you have to resubmit the whole sitemap?
- daimyoyo 16y agoMy question would be "does SO create contet at that rate?" It seems to me that google need not index your site faster than you're creating things for it to see. Is there a way to vary how often google indexes you with how often your uses create content automatically?
- simonw 16y agoGoogle is constantly re-indexing old pages, so the rate of new content creation isn't that big a factor in the crawl rate (though I imagine it does cause Google to ramp up their crawling rates if they aren't already running at the maximum).
- xentronium 16y agoYes, google webmaster tools include a multiposition slider for crawl rate. It's on the screenshot at the original answer.
- daimyoyo 16y agoOops. Didn't see it there. Sorry.
- efsavage 16y agoThis isn't exactly abnormal. SO is a big site with alot of fresh content, I'm guessing google indexes many thousands of sites at that rate. What's surprising to me is that it surprises them.
- vladd 16y agoIndeed. There are only 86K seconds in a day, or 31.5 million seconds in a year. Even if you assume that Google refreshes a page only a couple of times per year (which is absurd in terms of their freshness), it still means that you cannot have more than a few million pages in the index without accepting on average multiple crawls per second from the bots.
- mryall 16y agoWell, the necessity of scraping that much content would depend on how often it is being updated. There must be only a few thousand active pages on Stack Overflow on any given day, so executing a million-plus page views per day seems a bit overkill. Google tends to reduce its crawl frequency for content that changes less frequently. Perhaps in SO's case, it struggles to identify recently changed content without crawling the entire site.
- kmontrose 16y agoSO gets well over 10k completely new posts (Q&As) on a typical day. A few thousand older posts are edited a day. Throwing comments into the mix more than doubles the number of "new things", and of course every question, answer, comment, or edit is displayed on multiple pages (generally, on the owning user's page). tl;dr - there's a lot more page churn than you might expect.
- deleted 16y ago[deleted]
- necro 16y agoHere is our 10/s crawl stats too. Thought I'd share to get a contrast. Though about half the page crawls per day. Note the page load times :) Oops, recent regression due to building an internal cloud. So you're right, not out of the ordinary. http://www.pinkbike.com/photo/6365090/ http://www.pinkbike.com/photo/6365090/
- PaulHoule 16y agoIt's either that or let most of their content be uncrawled and show up on efreedom.com instead.
- wpeterson 16y agoI was surprised how much of a load webcrawlers (Google, Bing, Yahoo, etc) imposed on us at PatientsLikeMe, the majority of which is Google. The "intelligent" rate limiting results in a very high rate of crawl for many sites. We added additional caching and manually lowered the crawl rate to address this at PatientsLikeMe.
- wtn 16y agoCare to add some real numbers for PatientsLikeMe? Your post reads like namedropping for PatientsLikeMe.
- tristanperry 16y ago10 requests per second doesn't sound majorly high. That's 36,000 pages per hour which whilst big, doesn't sound too high, especially for a site as popular as SO (Alexa puts it at 137rd most popular site; granted, Alexa isn't the most accurate).
- reitzensteinm 16y agoThis is addressed in the post - apparently it's hitting pages that haven't been accessed in a while, starting background tasks - but it still seems odd to me. I'd have expected a huge amount of Stack Overflow's traffic to come from long tail searches, which should be basically the same thing. Excerpt for the lazy: "and when Google hits thousands of pages in a few minutes, that can kick off a lot of background work, such as rebuilding related questions. Not expensive by itself, but when multiplied by a hundred at once.. can be quite painful."
- ultrasaurus 16y agoThe rules are that you can't send google different page content than regular browsers, but there's no reason they have to run all the background processes on googlebot requests -- can't they just send it the most recent cached version?
- eli 16y agoNot a bad idea, but seems like it would be tricky to get right. You do kinda want Google to have the most recent version of a page, all other things being equal.
- ultrasaurus 16y agoAgreed, there's a sweet spot somewhere though, and it might not be the same for googlebot as a regular viewer).
- tomkarlo 16y agoWhy not put the cached content in the page by default, then do an update via AJAX only in the case where the cached version is old? That way it's not triggered for crawlers. It's probably secondary content anyway.
- anonymousDan 16y agoThe interesting thing to know would be how much more efficient a push-based indexing approach would be instead of the current pull-based model. If frequently updated sites could push change notifications to google it would solve this problem. However, I'm not sure how google could trust such sites not to overload its own servers.
- Zakuzaa 16y agoSitemaps?
- mryall 16y agoYeah, Google supports the sitemaps standard but that doesn't really cater for content as dynamic as Stack Overflow's. The last-updated format is a day rather than a timestamp, for example, making it useless for very-frequently updated content. http://www.sitemaps.org/protocol.php http://www.sitemaps.org/protocol.php
- gregable 16y agoThis is true, but the big win is being able to cull out a list of URLs that have not been updated and thus do not need recrawling.
- Jgrubb 16y agoI was searching around for some SVG radial background gradient something or other that I wasn't sure even existed a few days ago and the top hit to come up was a SO question that had been asked 7 hours before. Answered my question, too. I was impressed.
- m_myers 16y agoWhat's funny is when someone asks a question and you think "Oh, I'll bet I could answer this with a little googling." And the top result turns out to be the question you're trying to answer.
- v21 16y agoWhat's frustrating is trying to Google a question, finding forum threads telling the original questioner to use Google, and refusing to answer the question.
- evilduck 16y agoI was trying to look up an error[1] regarding Padrino the other day and someone had asked about the same problem I was having. The exact question I typed in had been asked on SO 6 hours before I encountered it and then deleted from SO, but Google had cached it. I was sad the top result led to a deleted question without an answer, but was impressed an exact match to my question was cached 2 minutes after it was asked and was the top result (hopefully it will be dropped from results soon since it leads to SO's 404 page now). [1] http://www.google.com/search?q=getting-nomethoderror-for-padrino-helper-methods http://www.google.com/search?q=getting-nomethoderror-for-pad...
- known 16y agoIsn't this against the spirit of PageRank's neutrality?
- ceejayoz 16y agoNo?
- pestaa 16y agoStack Overflow is almost at 10 questions asked per second, so where does the surprise come from? Honestly, I'd have guessed a lot higher number.
- joshu 16y agoDelicious used to get slammed by crawlers, too. 10 qps isn't that bad. I remember some ad network launched using delicious widgets as their text ads platform that hit us for 25 qps sustained.
- jacques_chester 16y agoI remember having a blog that was crushed by a sudden, unthrottled interest from Baidu.
- kposehn 16y agoWonder if anyone noticed how crazy their engineers are: http://blog.serverfault.com/post/performance-tuning-intel-nics/ http://blog.serverfault.com/post/performance-tuning-intel-ni...
- moe 16y agoI'd substitute "crazy" with "inexperienced" for this particular anecdote. The intel e1000 family is the safe default for NICs and using something else without an explicit reason is just not a good idea.
- plainOldText 16y agoWhat's interesting to me is that the crawlers suck on average 5GB of data per day (according to their graphs) :)