5 ms·

Nope. Multiple small servers. 1) you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need t

Nope. Multiple small servers. 1) you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. 2) having multiple small servers allows us to spread our service into different availability zones 3) multiple small servers allows us to do rolling deploys without bringing down our entire service 4) once we use the multiple small servers approach it’s easy to scale up and down our compute by adding or removing machines. Having one server it’s difficult to scale up or down without buying more machines. Small servers we can add incrementally but with the large server approach scaling up requires downtime and buying a new server.

Nextgrid 4y ago
> you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. Managing a handful of big servers can be done manually if needed - it's not pretty but it works and people have been doing it just fine before the cloud came along. If you intentionally plan on having dozens/hundreds of small servers, manual management becomes unsustainable and now you need a control plane such as Kubernetes, and all the complexity and failure modes it brings. > having multiple small servers allows us to spread our service into different availability zones So will 2 big servers in different AZs (whether cloud AZs or old-school hosting providers such as OVH). > multiple small servers allows us to do rolling deploys without bringing down our entire service Nothing prevents you from starting multiple instances of your app on one big server nor doing rolling deploys with big bare-metal assuming one server can handle the peak load (so you take out your first server out of the LB, upgrade it, put it back in the LB, then do the same for the second and so on). > once we use the multiple small servers approach it’s easy to scale up and down our compute by adding or removing machines. Having one server it’s difficult to scale up or down without buying more machines. Small servers we can add incrementally but with the large server approach scaling up requires downtime and buying a new server. True but the cost premium of the cloud often offsets the savings of autoscaling. A bare-metal capable of handling peak load is often cheaper than your autoscaling stack at low load, therefore you can just overprovision to always meet peak load and still come out ahead.
- SoftTalker 4y ago
  I manage hundreds of servers, and use Ansible. It's simple and it gets the job done. I tried to install Kubernetes on a cluster and couldn't get it to work. I mean I know it works, obviously, but I could not figure it out and decided to stay with what works for me.
  - deleted 4y ago
    [deleted]
  - eastbound 4y ago
    But it’s specific, and no-one will want to take over your job. The upside of a standard AWS CloudFormation file is that engineers are replaceable. They’re cargo-cult engineers, but they’re not worried for their career.
    - Nextgrid 4y ago
      > But it’s specific, and no-one will want to take over your job. It really depends what's on the table. Offer just half of the cost savings vs an equivalent AWS setup as a bonus (and pocket the other half) and I'm sure you'll find people who will happily do it (and you'll be happy to pocket the other half). For a lot of companies even just half of the cost savings would be a significant sum (reminds me of an old client who spent thousands per month on an RDS cluster that not only was slower than my entry-level MacBook, but ended up crapping out and stuck in an inconsistent state for 12 hours and required manual intervention from AWS to recover - so much for managed services - ended up restoring a backup but I wish I could've SSH'd in and recovered it in-place). As someone who uses tech as a means to an end and is more worried about the output said tech produces than the tech itself (aka I'm not looking for a job nor resume clout nor invites to AWS/Hashicorp/etc conferences, instead I bank on the business problems my tech solves), I'm personally very happy to get my hands dirty with old-school sysadmin stuff if it means I don't spend 10-20x the money on infrastructure just to make Jeff Bezos richer - my end customers don't know nor care either way while my wallet appreciates the cost savings.
zhte415 4y ago
It completely depends on what you doing. This was pointed out in the first paragraph of the article: > By thinking about the real operational considerations of our systems, we can get some insight into whether we actually need distributed systems for most things.
ericd 4y ago
On a big server, you would probably be running VMs rather than serving directly. And then it becomes easy to do most of what you're talking about - the big server is just a pool of resources from which to make small, single purpose VMs as you need them.
- Koshkin 4y ago
  Why VMs when you can use containers?
  - cestith 4y ago
    Why containers when you can use unikernel applications?
    - Koshkin 4y ago
      But can unikernel applications share a big server (without themselves running inside VMs)?
      - cestith 4y ago
        Unikernels are VM guests. They just need a hypervisor, not to run inside another VM.
    - mixmastamyk 4y ago
      Better support when at least in the neighborhood of the herd.
  - ericd 4y ago
    If you prefer those, go for it. I like my infra tech to be about as boring and battle tested as I can get it without big negatives in flexibility.
    - Koshkin 4y ago
      In theory, VMs should only be needed to run different OSes on one big box. Otherwise, what should have sufficed (speaking of what I 'prefer') is a multiuser OS that does not require additional layers to ensure security and proper isolation of users and their work environments from each other. Unfortunately, looks like UNIX and its descendants could not deliver on this basic need. (I wonder if Multics had something of a better design in this regard.)
rubiquity 4y ago
The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly never happens. To answer your points: 1) Not necessarily. You can stream data backups to remote storage and recover from that on a new single server as long as that recovery fits your Recovery Time Objective (RTO). 2) What's the benefit of multiple AZs if the SLA of a single AZ is greater than your intended availability goals? (Have you checked your provider's single AZ SLA?) 3) You can absolutely do rolling deploys on a single server. 4) Using one large server doesn't mean you can't compliment it with smaller servers on an as-needed basis. AWS even has a service for doing this. Which is to say: there aren't any prescriptions when it comes to such decisions. Some businesses warrant your choices, the vast majority do not.
- dvfjsdhgfv 4y ago
  > The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. It will die out naturally once people realize how much the times have changed and that the old solutions based on weaker hardware are no longer optimal.
- bombcar 4y ago
  > Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly never happens. Nobody wants to admit that their business or their department actually has a SLA of "as soon as you can, maybe tomorrow, as long as it usually works". So everything is pretend-engineered to be fifteen nines of reliability (when in reality it sometimes explodes because of the "attempts" to make it robust). Being honest about the actual requirements can be extremely helpful.
  - bob1029 4y ago

Nextgrid 4y ago

> you need to get over the hump and build in multiple servers into your architecture from the get go (the author says you need two servers minimum), so really we are talking about two big servers. Managing a handful of big servers can be done manually if needed - it's not pretty but it works and people have been doing it just fine before the cloud came along. If you intentionally plan on having dozens/hundreds of small servers, manual management becomes unsustainable and now you need a control plane such as Kubernetes, and all the complexity and failure modes it brings. > having multiple small servers allows us to spread our service into different availability zones So will 2 big servers in different AZs (whether cloud AZs or old-school hosting providers such as OVH). > multiple small servers allows us to do rolling deploys without bringing down our entire service Nothing prevents you from starting multiple instances of your app on one big server nor doing rolling deploys with big bare-metal assuming one server can handle the peak load (so you take out your first server out of the LB, upgrade it, put it back in the LB, then do the same for the second and so on). > once we use the multiple small servers approach it’s easy to scale up and down our compute by adding or removing machines. Having one server it’s difficult to scale up or down without buying more machines. Small servers we can add incrementally but with the large server approach scaling up requires downtime and buying a new server. True but the cost premium of the cloud often offsets the savings of autoscaling. A bare-metal capable of handling peak load is often cheaper than your autoscaling stack at low load, therefore you can just overprovision to always meet peak load and still come out ahead.

SoftTalker 4y ago
I manage hundreds of servers, and use Ansible. It's simple and it gets the job done. I tried to install Kubernetes on a cluster and couldn't get it to work. I mean I know it works, obviously, but I could not figure it out and decided to stay with what works for me.
- deleted 4y ago
  [deleted]
- eastbound 4y ago
  But it’s specific, and no-one will want to take over your job. The upside of a standard AWS CloudFormation file is that engineers are replaceable. They’re cargo-cult engineers, but they’re not worried for their career.
  - Nextgrid 4y ago
    > But it’s specific, and no-one will want to take over your job. It really depends what's on the table. Offer just half of the cost savings vs an equivalent AWS setup as a bonus (and pocket the other half) and I'm sure you'll find people who will happily do it (and you'll be happy to pocket the other half). For a lot of companies even just half of the cost savings would be a significant sum (reminds me of an old client who spent thousands per month on an RDS cluster that not only was slower than my entry-level MacBook, but ended up crapping out and stuck in an inconsistent state for 12 hours and required manual intervention from AWS to recover - so much for managed services - ended up restoring a backup but I wish I could've SSH'd in and recovered it in-place). As someone who uses tech as a means to an end and is more worried about the output said tech produces than the tech itself (aka I'm not looking for a job nor resume clout nor invites to AWS/Hashicorp/etc conferences, instead I bank on the business problems my tech solves), I'm personally very happy to get my hands dirty with old-school sysadmin stuff if it means I don't spend 10-20x the money on infrastructure just to make Jeff Bezos richer - my end customers don't know nor care either way while my wallet appreciates the cost savings.

zhte415 4y ago

It completely depends on what you doing. This was pointed out in the first paragraph of the article: > By thinking about the real operational considerations of our systems, we can get some insight into whether we actually need distributed systems for most things.

ericd 4y ago

On a big server, you would probably be running VMs rather than serving directly. And then it becomes easy to do most of what you're talking about - the big server is just a pool of resources from which to make small, single purpose VMs as you need them.

Koshkin 4y ago
Why VMs when you can use containers?
- cestith 4y ago
  Why containers when you can use unikernel applications?
  - Koshkin 4y ago
    But can unikernel applications share a big server (without themselves running inside VMs)?
    - cestith 4y ago
      Unikernels are VM guests. They just need a hypervisor, not to run inside another VM.
  - mixmastamyk 4y ago
    Better support when at least in the neighborhood of the herd.
- ericd 4y ago
  If you prefer those, go for it. I like my infra tech to be about as boring and battle tested as I can get it without big negatives in flexibility.
  - Koshkin 4y ago
    In theory, VMs should only be needed to run different OSes on one big box. Otherwise, what should have sufficed (speaking of what I 'prefer') is a multiuser OS that does not require additional layers to ensure security and proper isolation of users and their work environments from each other. Unfortunately, looks like UNIX and its descendants could not deliver on this basic need. (I wonder if Multics had something of a better design in this regard.)

rubiquity 4y ago

The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly never happens. To answer your points: 1) Not necessarily. You can stream data backups to remote storage and recover from that on a new single server as long as that recovery fits your Recovery Time Objective (RTO). 2) What's the benefit of multiple AZs if the SLA of a single AZ is greater than your intended availability goals? (Have you checked your provider's single AZ SLA?) 3) You can absolutely do rolling deploys on a single server. 4) Using one large server doesn't mean you can't compliment it with smaller servers on an as-needed basis. AWS even has a service for doing this. Which is to say: there aren't any prescriptions when it comes to such decisions. Some businesses warrant your choices, the vast majority do not.

dvfjsdhgfv 4y ago
> The line of thinking you follow is what is plaguing this industry with too much complexity and simultaneously throwing away incredible CPU and PCIe performance gains in favor of using the network. It will die out naturally once people realize how much the times have changed and that the old solutions based on weaker hardware are no longer optimal.
bombcar 4y ago
> Any technical decisions about how many instances to have and how they should be spread out needs to start as a business decision and end in crisp numbers about recovery point/time objections, and yet somehow that nearly never happens. Nobody wants to admit that their business or their department actually has a SLA of "as soon as you can, maybe tomorrow, as long as it usually works". So everything is pretend-engineered to be fifteen nines of reliability (when in reality it sometimes explodes because of the "attempts" to make it robust). Being honest about the actual requirements can be extremely helpful.
- bob1029 4y ago