6 ms·
StackExchange Performance Stats
- bzb3 6y agoThey should rewrite everything in python so they need 10 times as much hardware. The economy is stagnating
- redleggedfrog 6y agoI couldn't tell but are those CPU percentages real time?
- vyrotek 6y agoAnd here's where they were in 2016 https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
- heipei 6y agoI've been patiently waiting on on update for this post and others on the Stackoverflow architecture and hardware on his blog. Nick, if you're reading this, we're dying for on updated post ;)
- p_l 6y agoIf you compare the components listed on the performance stats, it seems to match the 2016 post pretty much 1:1. I wouldn't be surprised if they just updated the hardware a bit and that was it.
- kburman 6y agoI have worked in a project which required double infrastructure of what SO is using and render page at > 200ms. Our db size was 1/4 of stackoverflow but we don't even have 0.1% of content was stackoverflow offers.
- GordonS 6y agoWhat stack were you using? The only time I've personally seen render times that high is with large, complex databases in bad need of indexes and/or restructuring, and with hybrid cloud stuff, where cloud services are using an on-prem API or database.
- f6v 6y agoDo live queries for frequently accessed data without caching and you can go into seconds to render a page. I once came across a Postgres full-text query without an index. The page took a minute to render, I kid you not. And that page was arguably crucial to internal operations.
- kburman 6y agoRails, but I won't say rails was any way responsible for all this. It was poor design choices from our side. It was a large database due to unnecessary complexity and we do have indexes but due to poor design it was hard to gain any significant performance gains from adding indexes.
- BossingAround 6y agoWow, it's a bit surprising that they use C# under the hood. I wonder what was the cause for that decision; back when I started learning Java, C# was said to be the slower of the two. Honestly, I never wanted to get into C# as much because typically, it also means getting into Windows (from personal experience, companies using C# are heavily invested in Windows, and I love using Linux for everything). Do companies nowadays develop and deploy C# on Linux boxes? Edit: Just a reaction to a number of responses, I know C# is now fully open source. That doesn't mean companies using C# develop using Linux and deploy on Linux. C# being open source means very little to me when all deployments are on a proprietary platform.
- easton 6y agoI think the major driver for it was that Joel Spolsky worked at Microsoft for a long time (among other things, he was a PM for Excel), and therefore knew the MS stack the best. For a long time, Stack Exchange ran on IIS + SQL Server, I don't know if that's true anymore.
- BossingAround 6y agoAh! That makes perfect sense. Interesting, thanks for the comment.
- randompwd 6y agoNot the case. Spolshy was not a coder. https://meta.stackexchange.com/a/122854 https://meta.stackexchange.com/a/122854 > The main reason, it was the development stack Jeff, Jarrod and Geoff knew best when they embarked on the mission. Jeff Atwood comments on linked answer also.
- GordonS 6y agoC# dev here, mainly working in enterprise size companies who are Microsoft shops. We usually deploy on Windows for on-prem services, and on Linux for cloud services (some on VMs, some on containers, some serverless). I'm more of a Windows guy for desktop, but always prefer Linux for server. Nowadays dotnet is a first class citizen on Linux and MacOS, through dotnet core. On performance, there has been a big focus on perf by the dotnet team over the past years - I'd be shocked if Java could beat dotnet on just about any metric. If you know what you're doing you can write allocation-free code, and you even have hardware intrinsics at your disposal - I recently ported a hashing algorithm to C# and got the performance pretty close to parity with native C code (almost memory speed). C# doesn't get much love on HN, but IMO it's a truely fantastic language.
- atdt 6y agoAt first I was surprised that they don't use edge caching: it seems that all requests get proxied to a backend server, which use just two redis servers as a shared cache. A global internet site can't get good performance this way: RTT between NYC and Sydney is >200ms, so even if your application is lightning fast, it's going to feel slow to users in many parts of the world. Digging around a bit, it seems that they do use Fastly for edge CDN: https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/ https://nickcraver.com/blog/2017/05/22/https-on-stack-overfl.... Still, it's an impressively lean stack.
- EE84M3i 6y agoPotentially they can't get much edge cache offload for HTML/API content because many people are logged in and/or they do page customization even for non-logged in sessions and pushing logic to the edge isn't worth it.
- atdt 6y agoProbably you're right! A webpagetest from Sydney shows pretty dismal performance: https://www.webpagetest.org/result/200801_FJ_dd5f2a76d77bcaaac4ee45d5967ef4e7/ https://www.webpagetest.org/result/200801_FJ_dd5f2a76d77bcaa... ~2 seconds for first contentful paint. That's pretty dismal, IMO, and shows the limitation of their current setup.
- ksec 6y ago>That's pretty dismal, IMO, and shows the limitation of their current setup. Well they cant beat the speed of light. What could they do ?
- lixtra 6y agoCache their content on different continents and live with only eventual consistency. But as long as there is no competition, why should they complicate things.
- tenken 6y ago
- jchook 6y agoSurprised to see such low peak ops/sec on HAProxy
- rogerdonut 6y agoI'm curious to know what you mean by this? It's not like it can't handle more that is just what their peak workload was. With only 18% CPU usage at peak it's clear that it could handle much more.
- jchook 6y agoI guess I'm mostly curious how they handle websocket load balancing.
- skunkworker 6y agoThis is all for a single site (location) though right? I wonder how many backup AZs they have and how they’re replicating across zones. It’s surprisingly less infrastructure then I was imagining.
- p_l 6y agoTwo datacenters, they used to have each site run in 2 not-completely-full server racks (or maybe 4, been long time since I read the blog post of their switch gear upgrade to 10 gigabit). AFAIK they still keep to two datacenters, and the second one is a backup. All of SO sites run from the same few racks.
- recuter 6y ago300-450 rps for the web server tier seems very low. I wonder why that is.
- f6v 6y ago9x300x60x60x24x30=6998400000
- recuter 6y agoSure, I meant I'm surprised one server handles so few requests per second each.
- ksec 6y agoIn a blog post a few years ago they explained they could have fitted all 3000+ Req/s in a single server, but its ~99% percentile would have been ~100ms+. So in the name of performance and sub 20ms rendering time they decided to it with more servers.
- f6v 6y agoI’m underwhelmed by the absence of 139 micro services(written by 11 one-pizza teams), each with its own database instance.
- simonebrunozzi 6y agoTwo-pizzas teams.
- deleted 6y ago[deleted]
- qes 6y agoI serve near StackOverflow levels of traffic. Similar # of page views and more than 5x the egress bandwidth. Also .Net. Also running a traditional monolith style system. One main SQL DB. Another for some logs. Redis. Blob storage. KISS. We have various miscellany and a dozen+ web apps for public website, internal admin, content editing, monitoring, on-the-fly image resizing, etc. but even the API's serving those live in one web app, and our high traffic data & content service is just one app. Nearly everything in one solution and repo. Tbf, we've been utilizing chromium more over the past year and I think we're going to have to split that out into its own service. We do a lot of data-driven image generation and used a couple .Net imaging libraries for that until last year. Since then we've been building new data-driven content solely in HTML/JS and screenshotting through chromium to generate static images (we serve the same content in animated HTML and JPG and used to build each separately).
- ksec 6y agoI have always thought StackExchange as the Gold Standard of CRUD Apps. I would be happy if Ruby Rails could do it in 3x the rendering time ( i.e slower ) with 2x Resources / Server.
- li4ick 6y agoI wonder about the future of Stackexchange, now that every community has it's own subreddit/discort/etc.
- Can_Not 6y agothose two things are terrible at the Q&A format, search, and SEO, so probably not much to wonder about.
- simonebrunozzi 6y agoSo, no cloud, no AWS. All in house?
- rolls-reus 6y agoIs stackoverflow essentially a monolith? I remember reading about the single database setup several years ago, I'm surprised that they have been able to keep scaling vertically.
- deleted 6y ago[deleted]
- algorithmsRcool 6y agoI think that is the thesis of StackOverflow's devs. You can vertically scale easier and much further than what most people think. No groud up re-architecture, no microservice complexity, no crazy high cloud bills for exotic high power VMs. Just upfront capx for your own big iron and the skill to run it.
- vii 6y agoAmazon is offering 24TB RAM high memory machines on AWS. The Stackoverflow setup is far away from Big Iron :) In terms of opportunities to add complexity: I'm surprised there is no mention of a data-platform or any model training for ranking, etc. To reduce components, it might be possible to combine ElasticSearch and MSSQL into just PostgreSQL which has awesome text indices. As MSSQL performs very well and presents a somewhat esoteric SQL dialect this could be an expensive project.
- zamadatix 6y agoThe specs seem to line up with their upgrade notes from 2015 so it was a bit more "big iron" when made. Not that it's exactly under stress at the moment by any means but that's just further proof big iron can get you really far.
- matt2000 6y agoI love to read about how StackExchange does things, it's such a stark contrast to the cloud maximalists. I'm not saying there aren't advantages to each camp, it's just nice to have a solid example on the "vertical scaling can actually get you pretty damn far" side of things.
- ed_elliott_asc 6y agoOne day stack overflow will be offline for weeks and then people will have an answer to this.
- zamadatix 6y agoIt's 20 servers, many of which are hot spares. What about self hosting that is going to take weeks and why hasn't it happened in the last decade?
- alkonaut 6y agoI knew about most of their architecture, but what puzzles me is: why are some servers vertical boxes and some horizontal?
- Youden 6y agoIt could be blade vs. regular rackmount but I think it's just a design choice since the other ways of arranging 9 web servers aren't as visually appealing.
- MuffinFlavored 6y ago11k peak SQL queries @ 15% peak CPU usage across how many cores/processors?
- jmaygarden 6y agoThere’s really no need for all of the speculation in this thread. Jeff Atwood documented the StackOverflow development process as it happened on his blog. Here’s the first announcement from 2008: https://blog.codinghorror.com/introducing-stackoverflow-com/ https://blog.codinghorror.com/introducing-stackoverflow-com/
- brainzap 6y agoWhat about the CDN?
- ed_elliott_asc 6y agoStack overflow is awesome Stack overflow runs on .net Stack overflow squeezes lots of performance out of very little hardware or cost Stack overflow doesn’t do that much Customers haven’t added feature after feature until it resembles a half dog half cow hybrid Stack overview developers spend time working out how to scale for their specific use case rather that theoretical use cases I honestly believe that by sticking to a singular narrow focus stack overflow has achieve magnificent things. It isn’t because of .net it is in spite of it. Well done stack overflow but unless you can stop the business or the users with extra requests or expanding the product you cannot replicate it.
- syspec 6y agoDidn't know stackexchange made such use of websockets, where are they using them?
- haneefmubarak 6y agoThey use websockets for notifications (upvotes, comments, etc).
- DonCopal 6y agoMuch performance! Yet they don't even have a button to select all of code in a codeblock.
- rurban 6y agoI'm esp. impressed about the C# Json library Jil, they are using. Many useful tricks. After the first deserialization it internally optimizes itself to speed up further deserializations of the same format.