11 ms·
StackExchange System Architecture
- noise 12y agoAm I the only one that finds it annoying to have each layer's stats in different units? We have: TB/mo, reqs/sec, queries/day, reqs/min, searches/day.
- juliangregorian 12y agoIt would also be interesting to know how much it costs them in Microsoft licensing fees.
- BrentOzar 12y ago> It would also be interesting to know how much it costs them in Microsoft licensing fees. They started with BizSpark, Microsoft's program for startups that gives you free licensing for a few years. After that, you pay maintenance on what you've got, and buy new things as you add new servers.
- bsg75 12y agoStackExchange is a well-known-ish case of "startups can run Windows too". I wonder if they get any leeway on licensing for the PR?
- zamalek 12y ago> I wonder if they get any leeway on licensing for the PR? For production? No. Development machines? They used to. The best you can hope for these days as a showcase ISV/startup is more direct access to devs, and you'll probably only get that if you are proofing a new system of theirs. Remember, the most likely gospel is that Microsoft is expensive to run - so dropping the licensing cost "for PR" is actually a bad idea.
- BrentOzar 12y ago> > I wonder if they get any leeway on licensing for the PR? > For production? No. Development machines? They used to. Development licensing for SQL Server (and most of MS's products) is done through the MSDN program, and it's dirt cheap, and doesn't require any leeway. Anybody can buy MSDN licensing. For production, everything is negotiable. It's not unusual to get special SQL Server licensing deals when you buy in bulk. (That's me walking the line between what I can say publicly, and what I can't.)
- thirsteh 12y agoShh, we don't talk about that!
- alexchamberlain 12y agoVery interesting overview. It shows that the majority of a (Q&A) site can be run on "just" 9 servers - we're not talking scaling to thousands of servers here. I'd be interested to know what DBMS they're running, and it's very interesting that they've moved the tag "indices" out to their own cluster. I'd also be interested to know whether they run out of 1 datacentre and how/if they mitigate that risk.
- adamnemecek 12y agohttp://meta.stackexchange.com/questions/10369/which-tools-and-technologies-are-used-to-build-the-stack-exchange-network http://meta.stackexchange.com/questions/10369/which-tools-an...
- alexchamberlain 12y agoTL;DR Microsoft stack and 2 datacentres: NJ and Oregon.
- ayrx 12y agoPlus they have a lot of spare capacity on the hardware. Even if traffic were to double they wouldn't need new servers to keep things running smoothly.
- lclarkmichalek 12y agoA lot of spare capacity on the CPU != a lot of spare capacity. They are most likely much more constrained wrt RAM and disk ops
- ayrx 12y agoI very much doubt that's the case for their web servers. Note how they only opted for 48GB on their web servers vs the much larger numbers on their DB servers?
- codeulike 12y agoI'd be interested to know what DBMS they're running MS SQL Server, latest version probably
- Kiro 12y agoWhat's a tag engine server?
- ayrx 12y agoLooks like a custom written service. I quote from http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda... The Tag Engine is entirely self-contained, which means not having to depend on an external service for very, very core functionality. It's a huge in-memory struct array structure that is optimized for SO use cases and precomputed results for heavily hit combinations. It's a simple windows service running on a few boxes working in a redundant team. CPU is about 2-5% almost always. Three boxes are not needed for load, just redundancy. If all those do fail at once, the local web servers will load the tag engine in memory and keep on going.
- nfm 12y agoIt always puts me on edge when people choose to write their web-app in a particular language or framework 'because performance'. The C# VM is by no means slow but the fact that StackExchange is running off 9 front-end servers under pretty light load is testament to the fact that for regular web app workloads, it's just premature optimization to pull out the 'performance' card.
- juliangregorian 12y agoI wonder why they don't use more varnish or other cache servers instead. I've worked on a number of content-driven sites on the same scale as SO and that's a much more common pattern.
- patelsan 12y agoMaybe because most of their stack seems to be running on windows and varnish doesn't run on windows?
- juliangregorian 12y agoSo? They use haproxy. This also doesn't mean they can't use whatever the windows equivalent is; I simply don't know what that would be.
- melling 12y agoMaybe they only need so few servers because they are using a performant language like C#? Would they need a lot more servers if they used Rails, Django, or PHP? In other words, an interpreted language. I'm not so sure your conclusion can be derived from their stack.
- nfm 12y agoYeah that's my point - if they were using a slower language they'd be clocking in at say 20 or 30 servers. Not remotely worth optimizing for saving that many servers at their scale.
- mjibson 12y agoNick Craver has many more details on his blog: http://nickcraver.com/blog/ http://nickcraver.com/blog/
- mjs 12y agoIn particular, http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-s...
- PhilipA 12y agoI remember a podcast with Scott Hansleman when they just used 1 server (an still served tons of req/s). I think it was this one: http://www.hanselman.com/blog/HanselminutesPodcast134StackOverflowUsesASPNETMVCJeffAtwoodAndHisTechnicalTeam.aspx http://www.hanselman.com/blog/HanselminutesPodcast134StackOv... If I recall correctly Hansleman was quite impressed with the performance they could get from just the one server.
- juliangregorian 12y agoA SO employee told me this was still true less than 3 years ago.
- Maarten88 12y agoAt the PDC conference 2008 Jeff Atwood RDP'd into the live production webserver that was Stackoverflow as a guest speaker in the presentation by Phil Haack on the then-new ASP.NET MVC framework, see http://channel9.msdn.com/blogs/pdc2008/pc21 http://channel9.msdn.com/blogs/pdc2008/pc21 ca 49:40 into the presentation.
- tsu 12y agohow slow was that then?
- y0ghur7_xxx 12y agoSo they have 185 req/s for all their Stack Exchange sites. Let's say half of that is on Stack Overflow. That's 93 req/s and they do it with one DB (and one hot standby) with 384GB of RAM. I don't know many sites that big, so I think that the majority of people that say "the DB does not scale" work all for online properties with more that hundreds of millions of users, or are optimizing for a problem they will never have.
- andrea_s 12y agoOn the other hand, the StackExchange workflow allows for very aggressive content caching - they probably wouldn't be able to maintain the same performances if they needed to generate content on a finer level. edit: a few additional details - from a few years ago, alas http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
- y0ghur7_xxx 12y agoThat DB does 343M queries/day. If the average page requires 10 queries to render, that's 397 pages/second the db would be able to return data for. This math is obviously an average, but even if you halve that number you are still very much above everything most of us web devs are ever going to build or even see.
- andrea_s 12y agoAbsolutely! I was not demeaning their expertise, just saying that their workflow is helpful in reaching such impressive performances with seemingly so little hardware...
- JanezStupar 12y agoWell as soon as you start fiddling with performance one of the first things you figure out that to eliminate the worst bottlenecks you need to simplify the workflow around them.
- themartorana 12y ago
- beck5 12y agoDoes anyone have more information on the websockets? I assume they all go through haproxy to their 9 web servers, which is a decent 29k sustained connections per server.
- icebraining 12y agoThis might help: http://brokenhaze.com/blog/2014/03/25/how-stack-exchange-gets-the-most-out-of-haproxy/ http://brokenhaze.com/blog/2014/03/25/how-stack-exchange-get...
- jilljennV 12y agoThis seems to be an orphaned page. If you click on “team”, “performance” disappears from the menu. Apparently, it got leaked :) http://meta.stackexchange.com/a/244932 http://meta.stackexchange.com/a/244932
- mwsherman 12y agoIt’s not a secret exactly, obvs it has a public URL. It’s quietly been there for a couple of months. We kinda wanted to have a 2.0 version of the page with live data before promoting it. This one is static with #’s we sampled at the time.
- guiomie 12y agoI see their programming stack mentions ASP.Net MVC. Are you guys planning to move to vNext? Which version of IIS are you guys using? Is F# something also used?
- Nick-Craver 12y agoYes, we'll move to vNext when it's a bit more stable (likely before RTM). We're already compiling with Roslyn for localization for example.
- johansch 12y agoWow 185 req/s :) . Why do you even need more than a single modern cpu? All that cacheability.
- samirageb 12y agoIt's probably worth noting a few things about StackExchange's uniqueness for those that may not know. StackExchange runs on physical hardware and they have spent a considerable amount of time optimizing for bare metal performance. Their team is unique in that they embraced hardware from the start vs. many teams today that want hardware abstracted away. They have more maintenance overhead around hardware management, but don't experience lost IOPs due to IaaS (AWS/Azure) overhead. Their current environment required overcoming tremendous technical hurdles on earlier versions of SQL Server (these might be general RDBMS limitations as well). Luckily they were able to get the Microsoft SQL team engaged to get them through this. Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and others) are highly respected members of the SQL & .NET community. It's easy to look at their setup and say "Wow, that's not a lot." and overlook the circumstances & talent required to achieve such an efficient system. I'm not sure many teams would pursue this architecture if they knew the effort (and luck) involved.
- BrentOzar 12y ago> Their current environment required overcoming tremendous technical hurdles on earlier versions of SQL Server Jeff Atwood's goal was always "performance is a feature," and that means rendering your pages faster than anybody else's. With that in mind, you're always going to have technical hurdles to overcome on any platform - because you want to get faster results than anybody else is getting. I remember when I first got involved, and Jeff told me something along the lines of, "This slow query runs in ~500ms, and we want it to run in ~50." That statement alone is a huge leap over what a lot of RDBMS users say - usually when people refer to a slow query, their unit of measurement is whole seconds, not milliseconds. They were serious about performance from the get-go. > Luckily they were able to get the Microsoft SQL team engaged to get them through this. Nothing against Microsoft - I love 'em dearly and make a great living with their tools - but Stack's success is much more due to their own internal team's dedication to performance. When we open bug requests with Microsoft, the time to resolution is typically measured in weeks or months. During that time, Stack Exchange's team has to come up with creative, impressive workarounds. They're the sharpest SQL Server users I know. > Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and others) are highly respected members of the SQL & .NET community. Awww, shucks, but I'm not brown-nosing Stack when I say that their current team is ridiculously good. Their Site Reliability Engineers know more about SQL Server than most career DBAs I know. > I'm not sure many teams would pursue this architecture if they knew the effort (and luck) involved. It sounds like you're implying that other architectures are faster by default and with less effort, and I would disagree there. I haven't seen any big sites (big being top-100-in-the-world[1] type stuff) where the persistence layer was set-it-and-forget-it. Scale - especially millisecond performance at scale - is seriously hard work on all platforms. [1] https://www.quantcast.com/top-sites https://www.quantcast.com/top-sites
- wuliwong 12y agoBy no means am I an architecture expert but I've wondered if these statistics would look much different if stackoverflow had more images and video?
- tinco 12y agoThey would simply have additional file servers, could be any amount depending on how many video/images. They would possibly have to be in separate datacenters though, since shipping large amounts of data over seas can be inefficient and possibly expensive. I'm not sure at what point that starts to be a necessity, they're big but it's still a niche market. And their content is very sparsely visited (i.e. it wouldn't happen often that 80% of their audience would view a single question).
- samirageb 12y agoThe stats that would be hit the most obviously would be bandwidth, followed by RAM. Storage requirements would change of course, but bandwidth would be absorbed by whatever CDN they're using, so traffic quantity stat would be different. Also, with images/videos, they're now sending more URLs/embeds/etc. along with data, so more characters = more RAM for caching and/or serving up on web servers. In theory, shouldn't be much more of a hit.
- curiously 12y agohow did they get servers with insane amount of ram?
- WatchDog 12y agoYou can get servers with much more ram than 384GB. Here is a relatively inexpensive motherboard with 24 DIMMs that will go up to 768GB. http://www.newegg.com/Product/Product.aspx?Item=N82E16813151260 http://www.newegg.com/Product/Product.aspx?Item=N82E16813151... There are other solutions to take it much further than that.
- curiously 12y agoholy crap! so no more need for multiple clusters of servers, just up the ram and disk space.
- iancarroll 12y agoThis is of course why there are two options, vertical and horizontal scaling. Eventually you have to stop upgrading one server and split it up.
- jhildings 12y agoWell it would be boring it this huge RAM filled server got some hardware error...
- deleted 12y ago[deleted]
- jhildings 12y agoDoesn't it feel like a bit small environment? I was imaging at least 3x more servers