5 ms·

Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to ru

by konjin 6y ago

Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to run the application, one as backup and one as load balancer. 99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes. So that's why you end up with a server farm starting from your one small server.

akkartik 6y ago
Notice that grandparent comment explicitly carved out production servers. If you can't tolerate downtime while you're being upgraded, you're production.
franciscop 6y ago
99.99% of the websites out there don't need 99.99% uptime (from a business point of view)
- klodolph 6y ago
  Once you get to 99.9% uptime you’re still less than 1 hour per month. I used to run apps for companies at work and it was standard operating procedure to have two app servers + load balancer for upgrades. Drain a server, upgrade it, boot it, check that it works, let it accept traffic. Repeat once more. Done. It may be true that most businesses don’t need the uptime, but I’d say it’s usually easier to give them high availability rather than field complaints when some random user tries to look something up during a maintenance window.
  - vanviegen 6y ago
    But where does the data sit? Would you be using some fancy multi-master synchronous replication for that, or is there a database server behind it?
    - klodolph 6y ago
      A pair of database servers (leader and hot spare) behind it, although some data was in-app and shared between the two app instances (or more; two was the minimum). I don’t think of this as particularly fancy or expensive, considering how much you have to pay for engineers and technicians. It was critical enough for >3 nines, and that means having a technician on-call. I saw this setup at two different companies I worked at, even though the tech stacks were completely different. Whenever I price out setups for similar requirements in cloud I almost always end up with basically the same setup, just on cloud VMs instead of dedicated servers.
ClumsyPilot 6y ago
Basically nothing I have ever used had 99.99% uptime except maybe the phone and emergency services. Why do you need this level of uptime unless life and limb depends on you? My bank has regular scheduled downtime, Steam/Dota has regular downtime, entire Microsoft datacenters had issues and had around a day of downtime, Lime, Uber, my mobile carrier - all had downtime or bugs or glitches that were equivalent to downtime. If you really need crazy uptime, then you must make sure you have no bugs, because bugs cause more downtime than any hardware failures. That means an approach to software development that isn't relying on flavour of the month js library.
- nautilus12 6y ago
  But we can just stick my flavor of the month js library on lambda and bobs your uncle
  - zaptheimpaler 6y ago
    who is bob?????
    - ben_w 6y ago
      The phrase “Bob’s your uncle” is an idiom, not a specific or placeholder person as per Alice & Bob: https://en.m.wikipedia.org/wiki/Bob's_your_uncle https://en.m.wikipedia.org/wiki/Bob's_your_uncle
tobib 6y ago
You don't need multiple servers to achieve what you described.
- konjin 6y ago
  Fine, upgrade the hypervisor, still need two machines.
jlokier 6y ago
You don't need two servers rented, even for 99.99% uptime. That 99.99% uptime is achievable with a single dedicated server, plus cloud as a cold backup and some monitoring to trigger spinning it up. You will spin up the cloud server when the dedicate server is down. It will cost, but only for brief periods of time. That's sometimes cheaper than two dedicated servers. I prefer more than 1 dedicated server, but it's definitely possible to get the 99.99% uptime with just 1. And in some ways the extra diversity of having a different kind of backup is useful, e.g. for burst scaling as well, and for peace of mind not relying on a single provider. Even better, you can (and should) run the applications in VMs, or in containers inside VMs. They perform well these days, and even a severe application crash won't take down the host. Then regular application upgrades don't need a separate host server. Instead you're spinning up and shutting down VMs. You can also migrate VMs between hosts, and this can be done live if you want. It takes some setting up to have working distributed storage and movable networking, but it's possible. However, you might decide not to bother with that, as starting and stopping instances and having fleets of instances managed by something like K8S is modern practice for other comforting reasons anyway. Then the only "big" upgrades are limited to host server upgrades and major network reconfigurations, both of which you will do rarely. You might do host server security upgrades (kernel etc) more often, but that costs the time of a reboot unless it goes wrong. If those are planned or automated, those "big" upgrades can be done with zero user-facing downtime except in the event of failure, in which case it's limited to the cold-spare startup time So provided you have the backups in place, maybe on the cloud as a cold-spare so you're not paying for it except when it's needed, you can still have that 99.99% uptime for approximately the cost of one server. You do not need a separate load balancer device for uptime specifically; only if load balancing is something you want anyway for other reasons. For seamless transitions between backend servers, if you have multiple servers running at a particular time, there are other methods. On occasions you might instantiate a cloud load balancer temporarily to cover downtime during a planned host upgrade that can't be routed around another way. Like with a cloud cold-spare server, you don't pay for it except when it's temporarily instantiated.

by konjin 6y ago

akkartik 6y ago

Notice that grandparent comment explicitly carved out production servers. If you can't tolerate downtime while you're being upgraded, you're production.

franciscop 6y ago

99.99% of the websites out there don't need 99.99% uptime (from a business point of view)

klodolph 6y ago
Once you get to 99.9% uptime you’re still less than 1 hour per month. I used to run apps for companies at work and it was standard operating procedure to have two app servers + load balancer for upgrades. Drain a server, upgrade it, boot it, check that it works, let it accept traffic. Repeat once more. Done. It may be true that most businesses don’t need the uptime, but I’d say it’s usually easier to give them high availability rather than field complaints when some random user tries to look something up during a maintenance window.
- vanviegen 6y ago
  But where does the data sit? Would you be using some fancy multi-master synchronous replication for that, or is there a database server behind it?
  - klodolph 6y ago
    A pair of database servers (leader and hot spare) behind it, although some data was in-app and shared between the two app instances (or more; two was the minimum). I don’t think of this as particularly fancy or expensive, considering how much you have to pay for engineers and technicians. It was critical enough for >3 nines, and that means having a technician on-call. I saw this setup at two different companies I worked at, even though the tech stacks were completely different. Whenever I price out setups for similar requirements in cloud I almost always end up with basically the same setup, just on cloud VMs instead of dedicated servers.

ClumsyPilot 6y ago

Basically nothing I have ever used had 99.99% uptime except maybe the phone and emergency services. Why do you need this level of uptime unless life and limb depends on you? My bank has regular scheduled downtime, Steam/Dota has regular downtime, entire Microsoft datacenters had issues and had around a day of downtime, Lime, Uber, my mobile carrier - all had downtime or bugs or glitches that were equivalent to downtime. If you really need crazy uptime, then you must make sure you have no bugs, because bugs cause more downtime than any hardware failures. That means an approach to software development that isn't relying on flavour of the month js library.

nautilus12 6y ago
But we can just stick my flavor of the month js library on lambda and bobs your uncle
- zaptheimpaler 6y ago
  who is bob?????
  - ben_w 6y ago
    The phrase “Bob’s your uncle” is an idiom, not a specific or placeholder person as per Alice & Bob: https://en.m.wikipedia.org/wiki/Bob's_your_uncle https://en.m.wikipedia.org/wiki/Bob's_your_uncle

tobib 6y ago

You don't need multiple servers to achieve what you described.

konjin 6y ago
Fine, upgrade the hypervisor, still need two machines.

jlokier 6y ago

You don't need two servers rented, even for 99.99% uptime. That 99.99% uptime is achievable with a single dedicated server, plus cloud as a cold backup and some monitoring to trigger spinning it up. You will spin up the cloud server when the dedicate server is down. It will cost, but only for brief periods of time. That's sometimes cheaper than two dedicated servers. I prefer more than 1 dedicated server, but it's definitely possible to get the 99.99% uptime with just 1. And in some ways the extra diversity of having a different kind of backup is useful, e.g. for burst scaling as well, and for peace of mind not relying on a single provider. Even better, you can (and should) run the applications in VMs, or in containers inside VMs. They perform well these days, and even a severe application crash won't take down the host. Then regular application upgrades don't need a separate host server. Instead you're spinning up and shutting down VMs. You can also migrate VMs between hosts, and this can be done live if you want. It takes some setting up to have working distributed storage and movable networking, but it's possible. However, you might decide not to bother with that, as starting and stopping instances and having fleets of instances managed by something like K8S is modern practice for other comforting reasons anyway. Then the only "big" upgrades are limited to host server upgrades and major network reconfigurations, both of which you will do rarely. You might do host server security upgrades (kernel etc) more often, but that costs the time of a reboot unless it goes wrong. If those are planned or automated, those "big" upgrades can be done with zero user-facing downtime except in the event of failure, in which case it's limited to the cold-spare startup time So provided you have the backups in place, maybe on the cloud as a cold-spare so you're not paying for it except when it's needed, you can still have that 99.99% uptime for approximately the cost of one server. You do not need a separate load balancer device for uptime specifically; only if load balancing is something you want anyway for other reasons. For seamless transitions between backend servers, if you have multiple servers running at a particular time, there are other methods. On occasions you might instantiate a cloud load balancer temporarily to cover downtime during a planned host upgrade that can't be routed around another way. Like with a cloud cold-spare server, you don't pay for it except when it's temporarily instantiated.