16 ms·
A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on: https://stackexchange.com/performance https://stack
by IndrekR 7y ago
A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on:
https://stackexchange.com/performance https://stackexchange.com/performance
This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it.
SO ranks #36 in Alexa right now:
https://www.alexa.com/siteinfo/stackoverflow.com https://www.alexa.com/siteinfo/stackoverflow.com
- panpanna 7y agoNote that they use C# and ASP.Net... (SO and DailyWTF are very much into Microsofts ecosystem)
- tyuioyjj 7y agoAFAIK they're using ASP.NET Core There's significant difference between ASP.NET and ASP.NET Core https://meta.stackexchange.com/questions/316278/the-road-to-net-core-please-help-stack-exchange-test-ef-core https://meta.stackexchange.com/questions/316278/the-road-to-...
- Merad 7y agoThey use Core today, but SO was using .Net long before Core existed.
- FpUser 7y agoAnything wrong about that?
- riyadparvez 7y agoOne thing to keep in mind that their work-load is very ready-heavy which eases things a lot when scaling the system. The same is true for Wikipedia. Scaling a write-heavy workload is way more complex than scaling a read-heavy workload.
- hliyan 7y agoThere is a significant amount of question submission, commenting and voting going on on SO.
- yawn 7y agoA slim subset of humans typing things is not what I would label "write heavy". Write heavy is more like 100k+ devices out in the field sending their current position every 10s. That's still very manageable, but requires some thoughtful design.
- Retric 7y ago10k simple updates per second is hardly high bandwidth. I knew people handling similar workloads on a 1U server 10 years ago.
- pushpop 7y agoIndeed and I literally was. And the back end was written in mod_perl (which isn’t exactly known for performance). Even back then 10k/s was our smallest service - both on income as well as traffic - so there wasn’t the incentive to rewrite it in a faster language.
- stefan_ 7y agoThe Raspberry Pi 3 in my cupboard without a heat sink on can do 80000 Redis write ops/sec.
- manigandham 7y agoSO content changes all the time. Votes, comments, moderation, edits, tagging, search and recommendations, etc. There are also real-time community features. It's not as simple as it seems.
- perl4ever 7y agoAt the risk of exposing my ignorance - look at all those "Peak 5%-20%" labels. Doesn't that mean they have a lot more than they need?
- deleted 7y ago[deleted]
- daxfohl 7y agoNick Craver went into that a bit here. https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-...
- vitus 7y agoThey have a lot more CPU than they need, but the challenge is often getting servers shaped appropriately (RAM x CPU x disk x network), especially if you're not building your own. Ultimately, their service is (most likely) not compute-bound.
- dahfizz 7y agoStack overflow hosts all (most?) Of their own baremetal servers in their own data center. Looking at the specs of the machines, they are actually pretty basic as far as servers go. A server isn't barely worth the cost of it's chassis and motherboard if you put less than 64 G ram and 24 CPUs in it. In other words, these are about the lowest specd proper servers you can get. So yeah, even their modest hardware is still overspecd for running their website.
- fnord77 7y ago> That means we transfer 55 TB data / month this isn't a lot. Helps that site is mostly text data. I know of a relatively small cloud security system that transfers petabytes/month to/from a handful of customers. 4 ingest pods, 8 pipeline pods, 7 time-series db servers, 2 sql servers
- manigandham 7y agoThe amount of data transferred is meaningless to compare. SO is a relatively complex site with dynamic content and real-time features. Compared to other similar sites like Reddit or Quora which are far slower yet running on more hardware, it shows what proper efficient architecture and code can do.
- foota 7y agoReddit threads are also much hotter for writing and have O(thousands) of replies.
- manigandham 7y agoSure Reddit has more write activity but I don't see the number of replies being such a big factor. SO questions have answers, comments, tags, related questions, and many other secondary data to load. Reddit is slow due to poor architecture and a terrible frontend.
- onlyrealcuzzo 7y agoOld Reddit isn't slow. It seems to mostly be the new frontend.
- shakna 7y agoOld Reddit is still slow, but not too bad. Reddit's new software stack is... Atrocious.
- decebalus1 7y ago> SO ranks #36 in Alexa right now It would rank way higher if they would be more webscale. It needs more Kubernetes, GraphQL and Golang /s
- draw_down 7y agoThis joke was old 10 years ago. Find a new one already.
- 29athrowaway 7y agoPlus a content delivery network.
- jhgg 7y agoSO is really not a great example of a high traffic site. 5500 req/sec of mostly read only traffic is not that crazy at all, and their hardware footprint is incredibly over-provisioned for the workload. I don't really think their example stands well here. For example, at work, our entire analytics ingest workload (HTTP) for a few hundred million users runs on 8 core VMs on GCP, written in Rust/Go, each node doing ~40k events/second. Our RTC infra sustains >1m PPS per node on 4 core 3.9ghz 2014-2017 xeons.
- deleted 7y ago[deleted]
- z0mbie42 7y agoHi, I'm curious because I plan to rewrite a Rust service to Go (development velocity is too slow) Which part of your service is in Rust amd which in Go? Do you think that if it would be only in Go it could sustain such a load?
- zozbot234 7y agoWhat specific ergonomics pitfalls did you experience in Rust? "Development velocity is too slow" is a bit vague; beyond the use of GC, which only really matters in specialized domains, there's not much reason to think that rewriting your service in Go would give you better 'development velocity'.
- z0mbie42 7y agoBy development velocity I mean implementing new features, from idea to deployment. Rust is slow to compile so it breaks my deep work when programming, also it costs me a lot in CI/CD. Also the Rust type system make implementing some things really hard. For example I wanted to implement json requests logging. It took me more than 1 day in Rust, less than 2 hour in go.
- kick 7y agoThere is, though. Go was designed to be easier to reason with than conventional ALGOL-derivatives, while Rust wasn't. To quote Rob Pike: The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. Go is aiming for the same niche that Python is. This has the accidental side effect of making it faster to write software in it than most of the ALGOL-derivatives. To quote Eric Raymond (sorry) on Python: When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features! Go and Python both allow for expression about as fast as you can type, by virtue of being designed for [children in Go's case, shell scripting in Python's case]. It's not a value judgement of Rust or anything, but Rust wasn't designed with the same goals in mind.
- fasteo 7y agoAny reason for such a low (<5%) average CPU usage? It seems like a waste of resources to me; that's assuming a "normal" CPU usage read that includes wait i/o time.
- deleted 7y ago[deleted]
- hubert1234 7y ago600k open websocket connections means that 600k people opened stackoverflow sites but dont click on anything right? Because they still only have 550req/s. Interesting how much more power you need just to keep track of state.