7 ms·

You're mixing together useful complexity with useless complexity. Plus at the very least, I'd be very careful about putting type systems into the same basket a

You're mixing together useful complexity with useless complexity. Plus at the very least, I'd be very careful about putting type systems into the same basket as Kubernetes. One is a basic language feature used offline and before deploying. The other is a highly complex interwoven web of tools that might take your systems offline if used incorrectly. Without Kubernetes, you need Debian and it's Apache and MySQL packages. It's called a LAMP stack and for many production deployments, that's good enough. Because without all that "cloud magic", a $50 per month sever running a bare metal OS is beyond overpowered for most web apps, so you can skip all the scaling exercises. And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. A feat so difficult, I'd like to mention, that Amazon Web Services or Heruko rarely manage to... Complexity has high costs. Just because you don't see Kubernetes' complexity now, doesn't mean you won't pay for it through reduced performance, increased bug surface, increased downtime, or additional configuration nightmares.

threeseed 5y ago
This argument is often made and is ridiculous. No one should or is using Kubernetes to run a simple LAMP stack. But if you have dozens of containers and want them to be manager in a consistent, secure, observable and maintainable way then Kubernetes is going to be a better solution than anything you build yourself.
- tmp_anon_22 5y ago
  > No one should or is using Kubernetes to run a simple LAMP stack. Yes they are. Some developer got all excited about the capabilities of k8s, and had an initial larger scope for a project, so they set it up with GKE or EKS, and it managed to provide just enough business value to barrow in like a tick that won't be going away for years. Developers get all excited for new shiny tools and chuck it into production all the time, particularly at smaller orgs.
  - threeseed 5y ago
    > initial larger scope for a project So they weren't trying to run a simple LAMP stack then.
    - naniwaduni 5y ago
      Trying and doing can be very different things.
  - debarshri 5y ago
    I have seen smaller RoR, Django or lampstack apps being deployed on kubernetes exactly for reasons you mentioned. It is often pitch as a silver bullet for the future.
    - michaelpb 5y ago
      When Boss says "my idea is gonna be HUGE, so make this go fast", you can either spend 4 hours optimize some DB queries, or you can spend 40+ hours in a broadly scoped "conversion" project and have a new thing to add to your resume, and then spend 4 hours optimizing some DB queries...
      - debarshri 5y ago
        One of the worst, bazooka in knife fight moment was when I interviewed for a company that was a medium scale direct to consumer streaming service that was built some custom resource to create one pod per user, their worst case scenario was where they would have around 1000 concurrent users.
        alephu5 5y ago
        I worked in a Python web-scraping team doing 3 million requests per day. Their solution to manage concurrency was also one pod per crawl, each with its own DB connection. It really struck me as crazy that everyone thought this was a good idea.
    - ryandrake 5y ago
      Strikes me as someone buying a massive SUV, to only use it to move one bag of groceries from the store every week. Some day in the future they might suddenly have a family of 6 to feed or have to haul a boat somewhere, so better get the big, overkill car.
      - clpm4j 5y ago
        This describes ~70% of adults in the southeastern US.
  - danjac 5y ago
    It tends to happen with startups I think because developers know the startup is likely not going to be around for long, or they will move on in a couple years anyway. So why not shoehorn whatever tech you want and have that skill in your resume? Of course now the startup will need to hire for that tech when you leave and the cycle continues...
  - jlaurend 5y ago
    Heh, as someone who came very close to doing this (i.e. using k8s for a LAMP-stack type app at a startup), it's not just "shiny object syndrome" driving people to do this. Here's what our progression looked like: 1. Start with basic LAMP app in git that's manually deployed to an EC2 instance 2. Add in CI / CD + CodeDeploy 3. Create a staging environment 4. Dockerize local environment to keep dev environments in sync and onboard easier (really, this part's a gamechanger for a small company) 5. Ok so now we have Docker for local dev environments but stage and prod are managed separately. Can we just run our Docker containers in stage / production? When I researched step 5, the options were basically k8s or Docker swarm but Docker swarm didn't seem battle tested (for prod). k8s was clearly a nightmare for a small team to maintain so we started looking into GKE / EKS -- but EKS was still in beta. Thus we punted. We've actually started using ECS for a newer project and I'd likely go that route for step 5 instead.
- dvtrn 5y ago
  I agree that you probably shouldn’t but if you think no one “is”, I’d point to my last job, an enterprise that went to k8s for a single-serving php service that reads PDFs. I recently asked a friend who still works there if anything else has been pushed to k8s since I left (6 months ago). The answer: no.
  - derfabianpeter 5y ago
    Sounds familiar.
- lamontcg 5y ago
  I love the way that the Kubernetes debate always immediately devolves into Kubernetes vs. DIY where Kubernetes is the obviously correct answer. Two groups of people shouting past each other.
  - FpUser 5y ago
    My argument is that much of web development now often involves insane amount of tooling infrastructure and complexity all for the sake of delivering on cloud, scalable, virtualized, dockerized and otherwise over-processed equivalent of "hello world". And 99% (rhetoric number of course) of needed solutions would be served by a single real server (rent one on Hetzner or wherever) with 0 need to ever upgrade.
    - crooked-v 5y ago
      At a previous company we had Series C financing and enough people to take over two floors of an office building... and our entire SaaS offering for hundreds of white-label business customers ran fine on a couple of big load-balanced EC2 VMs and one big Postgres database with some read replicas.
      - patmorgan23 5y ago
        Yep, HA clusters/load balancing has been around for a long time and will get you pretty far even in some pretty large environments. Hell my companies main LOB app is running off a single Microsoft SQL serve(I know, I know, we have a lot of tech debt we're working through atm and it's on the list to set up HA). The longest downtime we've had was when we had to take it down to migrate it to our new hardware cluster and that was because the data transfer took 20 hours over a 1 GB port.
    - refenestrator 5y ago
      Another interesting angle about the single real server is the reliability.. it'll eventually break from some physical cause, that happens, it hurts your uptime. But sometimes with the big microservice/kubernetes solution, the configuration space is so big that miscommunications/mistakes could potentially take you down for more hours/year than the downsides of being single-hosted would. So now you invested all of that effort and for what? One company I worked for wrote a distributed scheduled jobs system that was immune to single-machine failures, it went down like 4 times in a year, messing up my team each time. I was like "guys, if we just provisioned one machine with cron and no failover, it would have better reliability".
- rodgerd 5y ago
  Alas, a lot of people are. One of the reasons there's such a backlash against k8s - other than contrarianism, which is always with us - is that there are quite a few people who have their job and hobby confused, and inflicted k8s (worse yet, raw k8s) on their colleagues not because of a carefully thought out assessment of its value, but because it is Cool and they would like to have it on their CV.
- michaelpb 5y ago
  > No one [...] is using Kubernetes to run a simple LAMP stack. Au contraire! This is very common. Probably some combo of resume-driven devops and "new shiny" excitement.
  - rileytg 5y ago
    or if you have a small team that must manage hundreds of these lamps… i started by simple wrapper around docker 8 or so years ago. over the years we’ve moved to k8s b/c it provided essentially the same api we had home grown. this reduced the LoC and moved the complexity of something like dynamic reverse proxy via nginx downstream into kubernetes ingress abstraction backed by nginx-ingress-controller.
    - danjac 5y ago
      That's not resume driven development though, that's picking the best tool for the job when you've outgrown the current solution.
debarshri 5y ago
Absolutely agree with you. I have seen the debate between accidental and necessary complexities very often. It actually depends upon stage of the organisation. In my opinion many devs in startups and smaller orgs try to accomodate the future expectations around product and create accidental complexities Accidental complexity becomes necessary complexity when an organisation scales out.
- kyleee 5y ago
  Another case of premature optimization, really
  - Aeolun 5y ago
    You really only need a single interview to find out companies care a lot about premature optimization. They’re always asking if you’ve worked with the latest tech stacks, never if you can set up a really well optimized nginx server.
    - hnzix 5y ago
      I see this as Resume Driven Development. Yak shaving is fun and at some point complex DevOps becomes necessary, but most CRUD apps benefit from a simple approach that allows high feature velocity. It's a balancing act between productivity and technical debt. I worked for one startup lead by a Java Architecture Astronaut, and working with the byzantine patterns and build systems made adding even simple features a morale-draining slog. It killed the product.
moonchild 5y ago
> You're mixing together useful complexity with useless complexity. Or, to channel Fred Brooks, essential and inessential complexity.
bob1029 5y ago
> You're mixing together useful complexity with useless complexity. > Complexity has high costs Complexity management is the central theme of building any large, valuable system. We would probably find that the more complex (and correct) a system, the more valuable it becomes on a relative basis to other competing solutions. The US tax code is a pretty damn good example of complexity intentionally taken to the extreme (for purposes of total market capture). We shouldn't be surprised to find other technology vendors framing problems & marketing their wares under similar pretenses. The best way to deal with complexity is to eliminate it or the conditions under which it must exist. For example, we made the engineering & product choice that says we do not ever intend to scale an instance of our application beyond the capabilities of a single server. Consider the implications of this constraint when reviewing how many engineers we actually need to hire, or if Kubernetes even makes sense. I think one of the biggest failings in software development is a lack of respect for the nature and impact of complexity. If we are serious about reducing or eliminating modes of complexity, we have to be willing to dig really deep and consider dramatic changes to the ways in which we architect these systems. I know its been posted to death on HN over the last ~48 hours, but Out of the Tar Pit is the best survey of complexity that I have seen in my career so far: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf
bostonsre 5y ago
> useless complexity Which items fall in this category?
- imwillofficial 5y ago
  Most of the time, K8s
  - bostonsre 5y ago
    Useless is a strong word in this context. Which parts or features of k8s are useless complexities? It's not like some random junior dev pulled a bunch of features out of a hat and implemented them. There was a ridiculous amount of thought put into its features and I can't think of a single complexity that is useless or even one that is useful but could be done in a more elegant /less complex way. Yes, it is complex, but there are lots of use cases where it is the most elegant and least complex solution. Yes, it definitely does not make sense to use it for a lamp stack deployed to one server, but there are use cases where it's a huge improvement (e.g. Spark on Hadoop is extremely complex and clunky when compared to spark on kubernetes).
    - fxtentacle 5y ago
      Like you said, if you're good with a LAMP stack and one server, then K8S is probably useless complexity. The issue is that most people nowadays have never worked with a bare metal Lamp server, so they grossly underestimate how large their company can grow before needing any distributed HA solution. I'd wager 90% of startups go bankrupt or are acquired before outgrowing a single Lamp server.
mwcampbell 5y ago
> And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box. A couple of problems with that: 1. What happens when you have to reboot to apply a kernel update? 2. The geographic location of that single box is itself a gap in redundancy. This is one thing I like about AWS and the other hyperscalers, with their regions that each have multiple data centers connected by a private network, with load balancers and other things spanning the region.
- FpUser 5y ago
  >"It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box" This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course. Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need. Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.
- aduitsis 5y ago
  Correct me, but I'd dare say admins have been rebooting their machines and running services without geographic redundancy for decades. Uptime was still very high.
- bob1029 5y ago
  This is a question that I have spent a ridiculous amount of time pondering. My conclusions so far are this: Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve. If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case. The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.
abraxas 5y ago
Couldn’t agree more! The crazy, ridiculous Rube Goldberg machines that are being strung together from those AWS components to solve the most mundane problems are getting ridiculous.

threeseed 5y ago

This argument is often made and is ridiculous. No one should or is using Kubernetes to run a simple LAMP stack. But if you have dozens of containers and want them to be manager in a consistent, secure, observable and maintainable way then Kubernetes is going to be a better solution than anything you build yourself.

tmp_anon_22 5y ago
> No one should or is using Kubernetes to run a simple LAMP stack. Yes they are. Some developer got all excited about the capabilities of k8s, and had an initial larger scope for a project, so they set it up with GKE or EKS, and it managed to provide just enough business value to barrow in like a tick that won't be going away for years. Developers get all excited for new shiny tools and chuck it into production all the time, particularly at smaller orgs.
- threeseed 5y ago
  > initial larger scope for a project So they weren't trying to run a simple LAMP stack then.
  - naniwaduni 5y ago
    Trying and doing can be very different things.
- debarshri 5y ago
  I have seen smaller RoR, Django or lampstack apps being deployed on kubernetes exactly for reasons you mentioned. It is often pitch as a silver bullet for the future.
  - michaelpb 5y ago
    When Boss says "my idea is gonna be HUGE, so make this go fast", you can either spend 4 hours optimize some DB queries, or you can spend 40+ hours in a broadly scoped "conversion" project and have a new thing to add to your resume, and then spend 4 hours optimizing some DB queries...
    - debarshri 5y ago
      One of the worst, bazooka in knife fight moment was when I interviewed for a company that was a medium scale direct to consumer streaming service that was built some custom resource to create one pod per user, their worst case scenario was where they would have around 1000 concurrent users.
      - alephu5 5y ago
        I worked in a Python web-scraping team doing 3 million requests per day. Their solution to manage concurrency was also one pod per crawl, each with its own DB connection. It really struck me as crazy that everyone thought this was a good idea.
  - ryandrake 5y ago
    Strikes me as someone buying a massive SUV, to only use it to move one bag of groceries from the store every week. Some day in the future they might suddenly have a family of 6 to feed or have to haul a boat somewhere, so better get the big, overkill car.
    - clpm4j 5y ago
      This describes ~70% of adults in the southeastern US.
- danjac 5y ago
  It tends to happen with startups I think because developers know the startup is likely not going to be around for long, or they will move on in a couple years anyway. So why not shoehorn whatever tech you want and have that skill in your resume? Of course now the startup will need to hire for that tech when you leave and the cycle continues...
- jlaurend 5y ago
  Heh, as someone who came very close to doing this (i.e. using k8s for a LAMP-stack type app at a startup), it's not just "shiny object syndrome" driving people to do this. Here's what our progression looked like: 1. Start with basic LAMP app in git that's manually deployed to an EC2 instance 2. Add in CI / CD + CodeDeploy 3. Create a staging environment 4. Dockerize local environment to keep dev environments in sync and onboard easier (really, this part's a gamechanger for a small company) 5. Ok so now we have Docker for local dev environments but stage and prod are managed separately. Can we just run our Docker containers in stage / production? When I researched step 5, the options were basically k8s or Docker swarm but Docker swarm didn't seem battle tested (for prod). k8s was clearly a nightmare for a small team to maintain so we started looking into GKE / EKS -- but EKS was still in beta. Thus we punted. We've actually started using ECS for a newer project and I'd likely go that route for step 5 instead.
dvtrn 5y ago
I agree that you probably shouldn’t but if you think no one “is”, I’d point to my last job, an enterprise that went to k8s for a single-serving php service that reads PDFs. I recently asked a friend who still works there if anything else has been pushed to k8s since I left (6 months ago). The answer: no.
- derfabianpeter 5y ago
  Sounds familiar.
lamontcg 5y ago
I love the way that the Kubernetes debate always immediately devolves into Kubernetes vs. DIY where Kubernetes is the obviously correct answer. Two groups of people shouting past each other.
- FpUser 5y ago
  My argument is that much of web development now often involves insane amount of tooling infrastructure and complexity all for the sake of delivering on cloud, scalable, virtualized, dockerized and otherwise over-processed equivalent of "hello world". And 99% (rhetoric number of course) of needed solutions would be served by a single real server (rent one on Hetzner or wherever) with 0 need to ever upgrade.
  - crooked-v 5y ago
    At a previous company we had Series C financing and enough people to take over two floors of an office building... and our entire SaaS offering for hundreds of white-label business customers ran fine on a couple of big load-balanced EC2 VMs and one big Postgres database with some read replicas.
    - patmorgan23 5y ago
      Yep, HA clusters/load balancing has been around for a long time and will get you pretty far even in some pretty large environments. Hell my companies main LOB app is running off a single Microsoft SQL serve(I know, I know, we have a lot of tech debt we're working through atm and it's on the list to set up HA). The longest downtime we've had was when we had to take it down to migrate it to our new hardware cluster and that was because the data transfer took 20 hours over a 1 GB port.
  - refenestrator 5y ago
    Another interesting angle about the single real server is the reliability.. it'll eventually break from some physical cause, that happens, it hurts your uptime. But sometimes with the big microservice/kubernetes solution, the configuration space is so big that miscommunications/mistakes could potentially take you down for more hours/year than the downsides of being single-hosted would. So now you invested all of that effort and for what? One company I worked for wrote a distributed scheduled jobs system that was immune to single-machine failures, it went down like 4 times in a year, messing up my team each time. I was like "guys, if we just provisioned one machine with cron and no failover, it would have better reliability".
rodgerd 5y ago
Alas, a lot of people are. One of the reasons there's such a backlash against k8s - other than contrarianism, which is always with us - is that there are quite a few people who have their job and hobby confused, and inflicted k8s (worse yet, raw k8s) on their colleagues not because of a carefully thought out assessment of its value, but because it is Cool and they would like to have it on their CV.
michaelpb 5y ago
> No one [...] is using Kubernetes to run a simple LAMP stack. Au contraire! This is very common. Probably some combo of resume-driven devops and "new shiny" excitement.
- rileytg 5y ago
  or if you have a small team that must manage hundreds of these lamps… i started by simple wrapper around docker 8 or so years ago. over the years we’ve moved to k8s b/c it provided essentially the same api we had home grown. this reduced the LoC and moved the complexity of something like dynamic reverse proxy via nginx downstream into kubernetes ingress abstraction backed by nginx-ingress-controller.
  - danjac 5y ago
    That's not resume driven development though, that's picking the best tool for the job when you've outgrown the current solution.

debarshri 5y ago

Absolutely agree with you. I have seen the debate between accidental and necessary complexities very often. It actually depends upon stage of the organisation. In my opinion many devs in startups and smaller orgs try to accomodate the future expectations around product and create accidental complexities Accidental complexity becomes necessary complexity when an organisation scales out.

kyleee 5y ago
Another case of premature optimization, really
- Aeolun 5y ago
  You really only need a single interview to find out companies care a lot about premature optimization. They’re always asking if you’ve worked with the latest tech stacks, never if you can set up a really well optimized nginx server.
  - hnzix 5y ago
    I see this as Resume Driven Development. Yak shaving is fun and at some point complex DevOps becomes necessary, but most CRUD apps benefit from a simple approach that allows high feature velocity. It's a balancing act between productivity and technical debt. I worked for one startup lead by a Java Architecture Astronaut, and working with the byzantine patterns and build systems made adding even simple features a morale-draining slog. It killed the product.

moonchild 5y ago

> You're mixing together useful complexity with useless complexity. Or, to channel Fred Brooks, essential and inessential complexity.

bob1029 5y ago

> You're mixing together useful complexity with useless complexity. > Complexity has high costs Complexity management is the central theme of building any large, valuable system. We would probably find that the more complex (and correct) a system, the more valuable it becomes on a relative basis to other competing solutions. The US tax code is a pretty damn good example of complexity intentionally taken to the extreme (for purposes of total market capture). We shouldn't be surprised to find other technology vendors framing problems & marketing their wares under similar pretenses. The best way to deal with complexity is to eliminate it or the conditions under which it must exist. For example, we made the engineering & product choice that says we do not ever intend to scale an instance of our application beyond the capabilities of a single server. Consider the implications of this constraint when reviewing how many engineers we actually need to hire, or if Kubernetes even makes sense. I think one of the biggest failings in software development is a lack of respect for the nature and impact of complexity. If we are serious about reducing or eliminating modes of complexity, we have to be willing to dig really deep and consider dramatic changes to the ways in which we architect these systems. I know its been posted to death on HN over the last ~48 hours, but Out of the Tar Pit is the best survey of complexity that I have seen in my career so far: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf

bostonsre 5y ago

> useless complexity Which items fall in this category?

imwillofficial 5y ago
Most of the time, K8s
- bostonsre 5y ago
  Useless is a strong word in this context. Which parts or features of k8s are useless complexities? It's not like some random junior dev pulled a bunch of features out of a hat and implemented them. There was a ridiculous amount of thought put into its features and I can't think of a single complexity that is useless or even one that is useful but could be done in a more elegant /less complex way. Yes, it is complex, but there are lots of use cases where it is the most elegant and least complex solution. Yes, it definitely does not make sense to use it for a lamp stack deployed to one server, but there are use cases where it's a huge improvement (e.g. Spark on Hadoop is extremely complex and clunky when compared to spark on kubernetes).
  - fxtentacle 5y ago
    Like you said, if you're good with a LAMP stack and one server, then K8S is probably useless complexity. The issue is that most people nowadays have never worked with a bare metal Lamp server, so they grossly underestimate how large their company can grow before needing any distributed HA solution. I'd wager 90% of startups go bankrupt or are acquired before outgrowing a single Lamp server.

mwcampbell 5y ago

> And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box. A couple of problems with that: 1. What happens when you have to reboot to apply a kernel update? 2. The geographic location of that single box is itself a gap in redundancy. This is one thing I like about AWS and the other hyperscalers, with their regions that each have multiple data centers connected by a private network, with load balancers and other things spanning the region.

FpUser 5y ago
>"It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box" This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course. Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need. Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.
aduitsis 5y ago
Correct me, but I'd dare say admins have been rebooting their machines and running services without geographic redundancy for decades. Uptime was still very high.
bob1029 5y ago
This is a question that I have spent a ridiculous amount of time pondering. My conclusions so far are this: Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve. If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case. The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.

abraxas 5y ago

Couldn’t agree more! The crazy, ridiculous Rube Goldberg machines that are being strung together from those AWS components to solve the most mundane problems are getting ridiculous.