19 ms·

Along with this, vertical scaling is severely underrated. You can do a lot and possibly everything ever for your company with vertical scaling. It would apply t

Along with this, vertical scaling is severely underrated. You can do a lot and possibly everything ever for your company with vertical scaling. It would apply to 99% of the companies or even more. Edit: Since people are confused, here is how StackOverflow handles of all of its web operations. If SO can run with this, so can your 0.33 req/minute app which is mostly doomed for failure. I am only half joking. StackOverflow architecture, current load (it will surprise you): https://stackexchange.com/performance https://stackexchange.com/performance Everytime you go to SO, it hits one of these 9 web servers and all data on SO sits on those 2 massive SQL servers. That's pretty amazing. I want to be clear though, Horizontal scaling has a place in companies that has a team of corporate lawyers. Big. And in many many other scenarios for ETL and backend microservices.

anonytrary 6y ago
Vertical scaling maybe works forever for the 99% of companies that are CRUD apps running a basic website. As soon as you add any kind of 2D or 3D processing like image, video, etc. you pretty much have to have horizontal scaling at some point. The sad truth is that your company probably won't be successful (statistically). You pretty never have to consider horizontal scaling until you have a few hundred thousand DAU.
- systemvoltage 6y ago
  Definitely. There is certainly a place for Horizontal scaling. Just wanted to highlight how underrated vertical scaling is and a good engineer would evaluate these scaling options with prudence and perspicacity, not cult behavior so often observed in software engineering circles.
- SilverRed 6y ago
  I imagine its still super simple to have one core app that handles most of the logic and then a job queue system that runs these high load jobs on worker machines. Much simpler than having everything split.
  - anonytrary 6y ago
    Worker machines sound like horizontal scaling to me.
    - SilverRed 6y ago
      Sure but its a massive amount simpler than some massively distributed microservice app where every component runs on multiple servers. Most of these vertical scaling examples given actually do use multiple servers but the core is one very powerful server.
- ComputerGuru 6y ago
  You don’t need to scale your application vertically even with media processing, you just need to distribute that chunk of the work, which is a lot easier (no state).
  - LockAndLol 6y ago
    It really depends. Handling streaming video service that does any kind of reprocessing of the data would probably be better off with horizontal scaling.
  - anonytrary 6y ago
    Like someone else said, distributing work across multiple machines is a form of horizontal scaling.
    - mhuffman 6y ago
      > Like someone else said, distributing work across multiple machines is a form of horizontal scaling. Sure, but it is the easy kind, when it comes to images or videos. Lambda, for example, can handle a huge amount of image processing for pennies per month and there is none of the additional machine baggage that comes with traditional horizontal scaling.
- dumb1224 6y ago
  I think somehow this is related to how business minded people think too. I went to a course where people learn to pitch their ideas to get funds but the basics of business simply did exist much among the technical people. One simple example (which I suspect most of the business do) is that you do all work either manually yourself or on your laptop while advise them as a resource-rich service. Only when you truly can not handle the demand then you may 'scale up' and turn your business into 'real' business. And there are plenty of tricks like this (as legally as possible).
sebmellen 6y ago
I think I agree, but what do you mean exactly? Just keep getting beefier servers as opposed to serverless junk?
- mavelikara 6y ago
  Not the OP, but yes, getting more powerful machines to run your program is what "vertical scaling" means (as opposed to running multiple copies of your program on similar-sized machines aka "horizontal scaling" ).
throwaway53453 6y ago
It might apply to 99% who have specific requirements, but the vast majority of internet companies need more. Deployments, N+1 redundancy, HA etc... are all valuable, even if some resources are going to waste.
- BeefySwain 6y ago
  > Deployments, N+1 redundancy, HA etc None of those things are mutually exclusive with vertical scaling? Having two identical servers for redundancy is doesn't mean you are scaling horizontally (assuming each can handle the load individually, nothing better than discovering that that assumption was incorrect in an outage).
  - TedDoesntTalk 6y ago
    Indeed, all of those things existed before docker.
soheil 6y ago
What is vertical scaling?
- toast0 6y ago
  Use bigger machines instead of more machines. There's always a limit on how big you can go, and a smaller limit on how big you should go, but eitherway it's pretty big. I wouldn't go past dual Intel Xeon, because 4P gets crazy expensive; I haven't been involved in systems work on Epyc, 1P might be a sensible limit, but maybe 2P makes sense for some uses.
  - soheil 6y ago
    If you have a single machine with 64 cores running 256 threads of your daemon that's considered vertical scaling? Odd definition
    - jjmarinho 6y ago
      If multiple cores/threads shouldn't be considered vertical scaling, what should? Overclocking a single-core processor from 2.8Ghz to 4.2Ghz can only take you so far after all...
    - yjftsjthsd-h 6y ago
      Yes, because it's still a single system image (one running OS with one process table).
    - SilverRed 6y ago
      Usually it involves having an insane amount of ram and keeping the entire DB in ram.
    - notreallyauser 6y ago
      'Scale up' vs 'scale out' are, to me, more intuitive terms for the same thing. Up/vertical are aligned, I guess
- The_rationalist 6y ago
SatvikBeri 6y ago
Several years ago, I was chatting with another engineer from a close competitor. He told me about how they'd set up a system to run hundreds of data processing jobs a day over a dozen machines, using docker, load balancing, a bunch of AWS stuff. I knew these jobs very well, they were basically identical for any company in the space. He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using. I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.
- cobookman 6y ago
  I once joked with a colleague that my sqlite3 install is faster than his Hadoop cluster for running a report across a multi-gig file. We benchmarked it, i was much, much faster. Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
  - movedx 6y ago
    It's all about the right tool for the job('s scale.)
  - Aeolun 6y ago
    At that point you can just get a bigger computer though.
    - ramraj07 6y ago
      No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry. I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
littlecranky67 6y ago
SO is always impressive - love that their redis servers with 256GB RAM peak at 2% CPU load :) SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. People seem to have lost the feeling what is needed to serve a couple of thousand concurrent users (For company internal usages which I specialize in, you hardly will get more users). Everyone thinks they are Google or Netflix. Meanwhile, SO runs on 1-2 Racks with an amount of server that would not even justify kubernetes or even docker.
- bscphil 6y ago
  The "1-2 racks" perspective is great too, really makes you think the old XKCD joke [1] about tripping over the power cable might not be that far wrong. ;-) [1] https://xkcd.com/908/ https://xkcd.com/908/
- sofixa 6y ago
  SO really isn't a great example, they have considerations most companies don't - Windows and SQL Server licensing. When shit like that is involved, scale out rarely seems like a better choice. It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.
- mschuster91 6y ago
  > SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop: - assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling. - no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically - no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore) - it's easy to set up new development instances in a CI/CD environment - generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal. I won't deny that there are issues though, especially if you're selfhosting: - you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own. - most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least - same if you're running anything with UDP, e.g. RTMP streaming - the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...
deleted 6y ago
[deleted]
hutrdvnj 6y ago
Thing is 99% of companies could run like SO if their software would be like SO. But if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed. It's very possible that you come to the conclusion that it is too complex to refactor for better vertical scaling. When your demand increases, then you simply buy another machine every now and then and spin up another instance of your monolith.
- zimpenfish 6y ago
  > if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed Last 15+ year old monolith I touched needed multiple machines to run because it was constrained by the database due to an insane homegrown ORM and poorly managed database schemas (and this is a common theme, I find.) Tuning the SQL, rejigging things like session management, etc., would have made it go a lot quicker on a lot fewer machines but management were insistent that it had to be redone as node microservices under k8s.
thefz 6y ago
Wow, thanks. This is the last place I expected to see C# mentioned. Very interesting!
- sumnole 6y ago
  It was mentioned in the article as being hard to build/test/deploy, but I disagree. Everything can be done in a few clicks using VS or Rider.
sofixa 6y ago
Stack Overflow have unique constraints ( Microsoft licensing) which make vertical scaling a cheaper option, and IMHO that's rarely the case.
antman 6y ago
Yes but Stackoverflow is a now mostly a graveyard of old closed questions, easily cached, I am only half joking. Most startup ideas today are a lot more interactive, so a SO model with two DBs would probably not serve them well. Horizontal scaling is not only for ETL and I am uncertain in why you say that it needs many lawyers.
mongol 6y ago
Might apply to 99 % of the companies but I doubt it applies to 99 % of the companies that HN readers work for.
ablekh 6y ago
> Everytime you go to SO, it hits one of these 9 web servers and all data on SO sits on those 2 massive SQL servers. That's pretty amazing. I don't find it amazing at all. Functionality-wise, StackOverflow is a very simple Web application. Moreover, SO's range of 300-500 requests per second is not a mind-blowing load. Even in 2014, a powerful enough single physical server (running a Java application) was able to handle 1M requests per second[1]. A bit later, in 2017, similar performance has been demonstrated on a single AWS EC2 instance, using Python (and a blazingly-fast HTTP-focused micro-framework Japronto), which is typically not considered a high-performance option for Web applications[2]. [1] https://www.techempower.com/blog/2014/03/04/one-million-http-rps-without-load-balancing-is-easy https://www.techempower.com/blog/2014/03/04/one-million-http... [2] https://www.freecodecamp.org/news/million-requests-per-second-with-python-95c137af319 https://www.freecodecamp.org/news/million-requests-per-secon...
- silvestrov 6y ago
  The amaziness is that the leadership allows it to be simple. This is such a great competitive advantage. Compare this to a leadership that thinks you absolutely must use Akamai for your 50 req/secs webserver. You end up with tons of complexity for no reason.
  - ablekh 6y ago
    Fair enough. Though not too surprising still, considering the original leadership of the company, one of whom (Joel Spolsky) is still on the board of directors. Having said that, the board's 5:4 VC-to-non-VC ratio looks pretty scary to me. But this is a different story ...
- mtberatwork 6y ago
  SO is a bit more complicated than returning a single character in a response. You can achieve high throughput with just about anything these days if you aren't doing any "work" on the server. 300-500 reqs/second is impressive for a web site/application with real-world traffic.
danpalmer 6y ago
SO is a great counter example to many over complicated setups, but they have a few important details going for them. > Everytime you go to SO, it hits one of these 9 web servers This isn't strictly true. Most SO traffic is logged out, most doesn't require strictly consistent data, most can be cached at the CDN. This means most page views should never reach their servers. This is obviously a great design! Caching at the CDN is brilliant. But there are a lot of services that can't be built like this.
- systemvoltage 6y ago
  CDN caches static assets. The request still goes to SO servers. Search goes to one of their massive Elastic Search servers. I’m not saying we should all use SO’s architecture, I am trying to shed light on what’s possible. YMMV obviously.
  - danpalmer 6y ago
    Are you an SO dev? I had thought I read about the use of CDNs and/or Varnish or something like that for rendered pages for logged out users? I don't want to correct you on your own architecture if you are!
    - systemvoltage 6y ago
      No, not a dev at SO. I am guessing what would be rather a standard use of CDN (hosting static assets, caching them geographically). What you're saying is probably right.
tcmb 6y ago
Genuine question, how is 9 web servers vertical scaling? And also, peak CPU usage of 12% means this is about 10x oversized for what is needed. Isn't it much better to only scale up when actually needed, mostly in terms of cost?
- elnygren 6y ago
  because they play in the major leagues where most teams have hundreds or thousands of servers, while they have those nine. Yes, there is some horizontal scaling, but the sheer amount of vertical scaling here is still mind blowing. I've run more servers in what were basically hobby projects compared to SO.
- jbverschoor 6y ago
  Probably 9 different geographical locations. Nothing to do with the actual load per server
nly 6y ago
Their database query to page request ratio is about 20:1. Seems like this should be lower.
brainzap 6y ago
people keep fapping to this, but stackoverflow is served read only to most users, and probably heavy cached.
weinzierl 6y ago
I totally agree with your main point and SO is kind of the perfect example. At the same time it is kind of the worst example because for one, to the best of my knowledge, their architecture is pretty much an outlier, and for another it is what it is for non-technical historical reasons. As far as I remember they started that way because they were on a Microsoft stack and Microsofts licensing policies were (are?) pretty much prohibitive for scaling out. It is an interesting question if they would design their system the same way if they'd the opportunity to start from scratch.
- systemvoltage 6y ago
  Most people responding here are nitpicking on whether SO’s architecture is the one. I wasn’t trying to imply that at all. I wanted to drive a point and SO is a good enough example to show that a massive company of the size of SO can run, so can your tiny app. Don’t scale prematurely. A lot can be done by reasonable vertical scaling. For one $120k/year Kubernetes infra engineer, you could pay for entire rack of beefy servers. Obviously YMMV. Discussion about SO and licensing details are distracting.
southerntofu 6y ago
Related, LetsEncrypt recently published a blogpost about their database servers and how they scale vertically: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...
bob1029 6y ago
We went all-in on vertical scaling with our product. We went so far we decided on SQLite because we were never going to plan to have a separate database server (or any separate host for that matter). 6 years later that assumption has still held very strong and yielded incredible benefits. The slowest production environment we run in today is still barely touched by our application during the heaviest parts of the day. We use libraries and tools capable of pushing millions of requests per second, but we typically only demand tens to hundreds throughout the day. Admitting your scale fits on a single host means you can leverage benefits that virtually no one else is even paying attention to anymore. These benefits can put entire sectors of our industry out of business if more developers were to focus on them.
- benbjohnson 6y ago
  Do you have any more details on your application? Sounds like your architecture choice worked out really well. I'm curious to hear more about it.
  - bob1029 6y ago
    Our technology choices for the backend are incredibly straightforward. The tricky bits are principally .NET Core and SQLite. One new technology we really like is Blazor, because their server-side mode of operation fits perfectly with our "everything on 1 server" grain, and obviates the need for additional front-end dependencies or APIs.
    - jjmod 6y ago
      how do you handle backup/replication for your sqlite server?
      - benbjohnson 6y ago
        I’m not the OP but I’m the author of an open source tool called Litestream[1] that does streaming replication of SQLite databases to AWS S3. I’ve found it to be a good, cheap way of keeping your data safe. [1]: https://litestream.io/ https://litestream.io/
gamedna 6y ago
Stack-overflow's use case has the benefit of being able to sit behind a Content Delivery Network (CDN) with a massive amount of infrastructure at the edge offloading much of the computational and database demands. This reduces the requirements of their systems dramatically. Given their experience in the segment, its plausible to expect they understand how to optimize their user-experience to balance out the hardware demands and costs as well.
calibas 6y ago
Looks like there's also two Redis servers with half a TB of RAM sitting in between web and SQL. I'm sure that's a huge load off the SQL server. https://meta.stackoverflow.com/a/306604/2489265 https://meta.stackoverflow.com/a/306604/2489265
louwrentius 6y ago
A ‘single’ big box with multiple terabytes of RAM can probably outperform many ‘horizontally scaled’ solutions. It all depend on the workload, but I feel that sometimes it’s more about being ’hip’ than being practical. https://yourdatafitsinram.net/ https://yourdatafitsinram.net/

anonytrary 6y ago

Vertical scaling maybe works forever for the 99% of companies that are CRUD apps running a basic website. As soon as you add any kind of 2D or 3D processing like image, video, etc. you pretty much have to have horizontal scaling at some point. The sad truth is that your company probably won't be successful (statistically). You pretty never have to consider horizontal scaling until you have a few hundred thousand DAU.

systemvoltage 6y ago
Definitely. There is certainly a place for Horizontal scaling. Just wanted to highlight how underrated vertical scaling is and a good engineer would evaluate these scaling options with prudence and perspicacity, not cult behavior so often observed in software engineering circles.
SilverRed 6y ago
I imagine its still super simple to have one core app that handles most of the logic and then a job queue system that runs these high load jobs on worker machines. Much simpler than having everything split.
- anonytrary 6y ago
  Worker machines sound like horizontal scaling to me.
  - SilverRed 6y ago
    Sure but its a massive amount simpler than some massively distributed microservice app where every component runs on multiple servers. Most of these vertical scaling examples given actually do use multiple servers but the core is one very powerful server.
ComputerGuru 6y ago
You don’t need to scale your application vertically even with media processing, you just need to distribute that chunk of the work, which is a lot easier (no state).
- LockAndLol 6y ago
  It really depends. Handling streaming video service that does any kind of reprocessing of the data would probably be better off with horizontal scaling.
- anonytrary 6y ago
  Like someone else said, distributing work across multiple machines is a form of horizontal scaling.
  - mhuffman 6y ago
    > Like someone else said, distributing work across multiple machines is a form of horizontal scaling. Sure, but it is the easy kind, when it comes to images or videos. Lambda, for example, can handle a huge amount of image processing for pennies per month and there is none of the additional machine baggage that comes with traditional horizontal scaling.
dumb1224 6y ago
I think somehow this is related to how business minded people think too. I went to a course where people learn to pitch their ideas to get funds but the basics of business simply did exist much among the technical people. One simple example (which I suspect most of the business do) is that you do all work either manually yourself or on your laptop while advise them as a resource-rich service. Only when you truly can not handle the demand then you may 'scale up' and turn your business into 'real' business. And there are plenty of tricks like this (as legally as possible).

sebmellen 6y ago

I think I agree, but what do you mean exactly? Just keep getting beefier servers as opposed to serverless junk?

mavelikara 6y ago
Not the OP, but yes, getting more powerful machines to run your program is what "vertical scaling" means (as opposed to running multiple copies of your program on similar-sized machines aka "horizontal scaling" ).

throwaway53453 6y ago

It might apply to 99% who have specific requirements, but the vast majority of internet companies need more. Deployments, N+1 redundancy, HA etc... are all valuable, even if some resources are going to waste.

BeefySwain 6y ago
> Deployments, N+1 redundancy, HA etc None of those things are mutually exclusive with vertical scaling? Having two identical servers for redundancy is doesn't mean you are scaling horizontally (assuming each can handle the load individually, nothing better than discovering that that assumption was incorrect in an outage).
- TedDoesntTalk 6y ago
  Indeed, all of those things existed before docker.

soheil 6y ago

What is vertical scaling?

toast0 6y ago
Use bigger machines instead of more machines. There's always a limit on how big you can go, and a smaller limit on how big you should go, but eitherway it's pretty big. I wouldn't go past dual Intel Xeon, because 4P gets crazy expensive; I haven't been involved in systems work on Epyc, 1P might be a sensible limit, but maybe 2P makes sense for some uses.
- soheil 6y ago
  If you have a single machine with 64 cores running 256 threads of your daemon that's considered vertical scaling? Odd definition
  - jjmarinho 6y ago
    If multiple cores/threads shouldn't be considered vertical scaling, what should? Overclocking a single-core processor from 2.8Ghz to 4.2Ghz can only take you so far after all...
  - yjftsjthsd-h 6y ago
    Yes, because it's still a single system image (one running OS with one process table).
  - SilverRed 6y ago
    Usually it involves having an insane amount of ram and keeping the entire DB in ram.
  - notreallyauser 6y ago
    'Scale up' vs 'scale out' are, to me, more intuitive terms for the same thing. Up/vertical are aligned, I guess
The_rationalist 6y ago

SatvikBeri 6y ago

Several years ago, I was chatting with another engineer from a close competitor. He told me about how they'd set up a system to run hundreds of data processing jobs a day over a dozen machines, using docker, load balancing, a bunch of AWS stuff. I knew these jobs very well, they were basically identical for any company in the space. He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using. I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.

cobookman 6y ago
I once joked with a colleague that my sqlite3 install is faster than his Hadoop cluster for running a report across a multi-gig file. We benchmarked it, i was much, much faster. Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
- movedx 6y ago
  It's all about the right tool for the job('s scale.)
- Aeolun 6y ago
  At that point you can just get a bigger computer though.
  - ramraj07 6y ago
    No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry. I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.

littlecranky67 6y ago

SO is always impressive - love that their redis servers with 256GB RAM peak at 2% CPU load :) SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. People seem to have lost the feeling what is needed to serve a couple of thousand concurrent users (For company internal usages which I specialize in, you hardly will get more users). Everyone thinks they are Google or Netflix. Meanwhile, SO runs on 1-2 Racks with an amount of server that would not even justify kubernetes or even docker.

bscphil 6y ago
The "1-2 racks" perspective is great too, really makes you think the old XKCD joke [1] about tripping over the power cable might not be that far wrong. ;-) [1] https://xkcd.com/908/ https://xkcd.com/908/
sofixa 6y ago
SO really isn't a great example, they have considerations most companies don't - Windows and SQL Server licensing. When shit like that is involved, scale out rarely seems like a better choice. It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.
mschuster91 6y ago
> SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop: - assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling. - no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically - no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore) - it's easy to set up new development instances in a CI/CD environment - generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal. I won't deny that there are issues though, especially if you're selfhosting: - you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own. - most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least - same if you're running anything with UDP, e.g. RTMP streaming - the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...

deleted 6y ago

[deleted]

hutrdvnj 6y ago

Thing is 99% of companies could run like SO if their software would be like SO. But if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed. It's very possible that you come to the conclusion that it is too complex to refactor for better vertical scaling. When your demand increases, then you simply buy another machine every now and then and spin up another instance of your monolith.

zimpenfish 6y ago
> if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed Last 15+ year old monolith I touched needed multiple machines to run because it was constrained by the database due to an insane homegrown ORM and poorly managed database schemas (and this is a common theme, I find.) Tuning the SQL, rejigging things like session management, etc., would have made it go a lot quicker on a lot fewer machines but management were insistent that it had to be redone as node microservices under k8s.

thefz 6y ago

Wow, thanks. This is the last place I expected to see C# mentioned. Very interesting!

sumnole 6y ago
It was mentioned in the article as being hard to build/test/deploy, but I disagree. Everything can be done in a few clicks using VS or Rider.

sofixa 6y ago

Stack Overflow have unique constraints ( Microsoft licensing) which make vertical scaling a cheaper option, and IMHO that's rarely the case.

antman 6y ago

Yes but Stackoverflow is a now mostly a graveyard of old closed questions, easily cached, I am only half joking. Most startup ideas today are a lot more interactive, so a SO model with two DBs would probably not serve them well. Horizontal scaling is not only for ETL and I am uncertain in why you say that it needs many lawyers.

mongol 6y ago

Might apply to 99 % of the companies but I doubt it applies to 99 % of the companies that HN readers work for.

ablekh 6y ago

> Everytime you go to SO, it hits one of these 9 web servers and all data on SO sits on those 2 massive SQL servers. That's pretty amazing. I don't find it amazing at all. Functionality-wise, StackOverflow is a very simple Web application. Moreover, SO's range of 300-500 requests per second is not a mind-blowing load. Even in 2014, a powerful enough single physical server (running a Java application) was able to handle 1M requests per second[1]. A bit later, in 2017, similar performance has been demonstrated on a single AWS EC2 instance, using Python (and a blazingly-fast HTTP-focused micro-framework Japronto), which is typically not considered a high-performance option for Web applications[2]. [1] https://www.techempower.com/blog/2014/03/04/one-million-http-rps-without-load-balancing-is-easy https://www.techempower.com/blog/2014/03/04/one-million-http... [2] https://www.freecodecamp.org/news/million-requests-per-second-with-python-95c137af319 https://www.freecodecamp.org/news/million-requests-per-secon...

silvestrov 6y ago
The amaziness is that the leadership allows it to be simple. This is such a great competitive advantage. Compare this to a leadership that thinks you absolutely must use Akamai for your 50 req/secs webserver. You end up with tons of complexity for no reason.
- ablekh 6y ago
  Fair enough. Though not too surprising still, considering the original leadership of the company, one of whom (Joel Spolsky) is still on the board of directors. Having said that, the board's 5:4 VC-to-non-VC ratio looks pretty scary to me. But this is a different story ...
mtberatwork 6y ago
SO is a bit more complicated than returning a single character in a response. You can achieve high throughput with just about anything these days if you aren't doing any "work" on the server. 300-500 reqs/second is impressive for a web site/application with real-world traffic.

danpalmer 6y ago

SO is a great counter example to many over complicated setups, but they have a few important details going for them. > Everytime you go to SO, it hits one of these 9 web servers This isn't strictly true. Most SO traffic is logged out, most doesn't require strictly consistent data, most can be cached at the CDN. This means most page views should never reach their servers. This is obviously a great design! Caching at the CDN is brilliant. But there are a lot of services that can't be built like this.

systemvoltage 6y ago
CDN caches static assets. The request still goes to SO servers. Search goes to one of their massive Elastic Search servers. I’m not saying we should all use SO’s architecture, I am trying to shed light on what’s possible. YMMV obviously.
- danpalmer 6y ago
  Are you an SO dev? I had thought I read about the use of CDNs and/or Varnish or something like that for rendered pages for logged out users? I don't want to correct you on your own architecture if you are!
  - systemvoltage 6y ago
    No, not a dev at SO. I am guessing what would be rather a standard use of CDN (hosting static assets, caching them geographically). What you're saying is probably right.

tcmb 6y ago

Genuine question, how is 9 web servers vertical scaling? And also, peak CPU usage of 12% means this is about 10x oversized for what is needed. Isn't it much better to only scale up when actually needed, mostly in terms of cost?

elnygren 6y ago
because they play in the major leagues where most teams have hundreds or thousands of servers, while they have those nine. Yes, there is some horizontal scaling, but the sheer amount of vertical scaling here is still mind blowing. I've run more servers in what were basically hobby projects compared to SO.
jbverschoor 6y ago
Probably 9 different geographical locations. Nothing to do with the actual load per server

nly 6y ago

Their database query to page request ratio is about 20:1. Seems like this should be lower.

brainzap 6y ago

people keep fapping to this, but stackoverflow is served read only to most users, and probably heavy cached.

weinzierl 6y ago

I totally agree with your main point and SO is kind of the perfect example. At the same time it is kind of the worst example because for one, to the best of my knowledge, their architecture is pretty much an outlier, and for another it is what it is for non-technical historical reasons. As far as I remember they started that way because they were on a Microsoft stack and Microsofts licensing policies were (are?) pretty much prohibitive for scaling out. It is an interesting question if they would design their system the same way if they'd the opportunity to start from scratch.

systemvoltage 6y ago
Most people responding here are nitpicking on whether SO’s architecture is the one. I wasn’t trying to imply that at all. I wanted to drive a point and SO is a good enough example to show that a massive company of the size of SO can run, so can your tiny app. Don’t scale prematurely. A lot can be done by reasonable vertical scaling. For one $120k/year Kubernetes infra engineer, you could pay for entire rack of beefy servers. Obviously YMMV. Discussion about SO and licensing details are distracting.

southerntofu 6y ago

Related, LetsEncrypt recently published a blogpost about their database servers and how they scale vertically: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...

bob1029 6y ago

We went all-in on vertical scaling with our product. We went so far we decided on SQLite because we were never going to plan to have a separate database server (or any separate host for that matter). 6 years later that assumption has still held very strong and yielded incredible benefits. The slowest production environment we run in today is still barely touched by our application during the heaviest parts of the day. We use libraries and tools capable of pushing millions of requests per second, but we typically only demand tens to hundreds throughout the day. Admitting your scale fits on a single host means you can leverage benefits that virtually no one else is even paying attention to anymore. These benefits can put entire sectors of our industry out of business if more developers were to focus on them.

benbjohnson 6y ago
Do you have any more details on your application? Sounds like your architecture choice worked out really well. I'm curious to hear more about it.
- bob1029 6y ago
  Our technology choices for the backend are incredibly straightforward. The tricky bits are principally .NET Core and SQLite. One new technology we really like is Blazor, because their server-side mode of operation fits perfectly with our "everything on 1 server" grain, and obviates the need for additional front-end dependencies or APIs.
  - jjmod 6y ago
    how do you handle backup/replication for your sqlite server?
    - benbjohnson 6y ago
      I’m not the OP but I’m the author of an open source tool called Litestream[1] that does streaming replication of SQLite databases to AWS S3. I’ve found it to be a good, cheap way of keeping your data safe. [1]: https://litestream.io/ https://litestream.io/

gamedna 6y ago

Stack-overflow's use case has the benefit of being able to sit behind a Content Delivery Network (CDN) with a massive amount of infrastructure at the edge offloading much of the computational and database demands. This reduces the requirements of their systems dramatically. Given their experience in the segment, its plausible to expect they understand how to optimize their user-experience to balance out the hardware demands and costs as well.

calibas 6y ago

Looks like there's also two Redis servers with half a TB of RAM sitting in between web and SQL. I'm sure that's a huge load off the SQL server. https://meta.stackoverflow.com/a/306604/2489265 https://meta.stackoverflow.com/a/306604/2489265

louwrentius 6y ago

A ‘single’ big box with multiple terabytes of RAM can probably outperform many ‘horizontally scaled’ solutions. It all depend on the workload, but I feel that sometimes it’s more about being ’hip’ than being practical. https://yourdatafitsinram.net/ https://yourdatafitsinram.net/