4 ms·

> Single node application systems are by far the most reliable and manageable from a business logic standpoint. Manageable? Maybe. Reliable? No. Most companies

> Single node application systems are by far the most reliable and manageable from a business logic standpoint. Manageable? Maybe. Reliable? No. Most companies don't need it but if a pipe bursts above the server room and now all of your medical records for your hospital are unavailable you're going to have a bad time. > At no point does spreading a problem across more than 1 computer make that problem easier to solve. I don't think anyone claimed it was easier (yet). The main thing people strive for is: 1. Dynamically reshaping your compute needs to match any needs. 2. Surviving failures of nodes or crashes in your application. 3. Managing all of the BS that goes with obtaining this (logs, etc). > Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable. I hope this isn't the case but this sounds like quite the death flag.

brian_cunnie 5y ago
I’ve only seen one burst pipe bring down a computer (Pitney Bowes, 1995), but I’ve witnessed several S3 outages.
- zthrowaway 5y ago
  You should spend a week working on a infra / ops team then.
  - Demiurge 5y ago
    I worked with multiple ops/infra teams, and have seen standard server rooms and advanced server rooms. They’re not supposed to have pipes leaking on the racks, that’s ridiculous. I know many stable redhat servers that didn’t need to reboot for years. I also maintain websites on hetzner servers that haven’t needed to reboot in years, under pretty high load. On the other hand, I got a hostnoc dedicated server and that thing went offline three times a month. So yeah, you have to know what you’re doing regardless of whether you’re building a server room or a k8s cluster.
    - mwcampbell 5y ago
      > I know many stable redhat servers that didn’t need to reboot for years. That's not actually something to brag about, as it implies they weren't receiving important kernel updates, particularly security updates.
      - bob1029 5y ago
        So, we do ~daily restarts on our customers' systems. We have a solid 12 hours per day to patch up a server before it needs to be in production again. That said. We communicate directly with other systems on the same premises that probably haven't been rebooted in over a decade. Building something that can run 10 years without downtime is not happenstance. It is very intentional and deliberate engineering.
      - fxtentacle 5y ago
        https://en.wikipedia.org/wiki/Kpatch https://en.wikipedia.org/wiki/Kpatch Allows you to patch the kernel without reboot.
      - Demiurge 5y ago
        As mentioned above, they were, both RedHat and Ubuntu now support kernel updates without restart.
    - gravypod 5y ago
      More modern data centers have liquid pumps available for low-cost cooling.
hnjst 5y ago
Completely aligned with my parent, day-to-day operations are often slightly more complex than making sure your server run your app properly, I'd argue that our services are increasingly dependent on others (in both directions, dependencies multiply and are more and more critical). That's also by interacting more with external entities that they bring more value. > The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. This part was what I felt deserved a counter-point though. Consensus is indeed at the core of the issue once you want distributed fault tolerance. However, I think you'll quickly hit two things with your approach: 1-2ms of latency, I fear that it may come with highly correlated failures on the "first tier of resilience". Moreover, the "second tier" being much farther, keeping them in consensus will imply harsh trade-offs. If you use synchronous consensus protocols, you'll slow down drastically the "first tier" (assuming you want consistency), if you go for the asynchronous replication (not consensus, this matters...) then the second tier can't really intervene on leader election or failover without risking a partition on a false positive (and if you try to be conservative there your RPO will suffer). If you're into these (fundamental) issues, I'd recommend Leslie Lamport's work (i.e. https://www.college-de-france.fr/site/en-martin-abadi/seminar-2011-05-11-11h00.htm https://www.college-de-france.fr/site/en-martin-abadi/semina... or http://www.lamport.org/ http://www.lamport.org/), the paper pointing a disappointing impossibility: https://dl.acm.org/doi/abs/10.1145/3149.214121 https://dl.acm.org/doi/abs/10.1145/3149.214121 and its generalization (https://dl.acm.org/doi/abs/10.1145/167088.167119 https://dl.acm.org/doi/abs/10.1145/167088.167119).
- bob1029 5y ago
  I appreciate the commentary. I feel most of this effectively boils down to "it depends". Clearly, there is no one-size-fits-all solution that can make everyone happy all the time, especially considering all the constraints & variables present across the entire vertical. Where your datacenters are geographically located is usually a big first step in even starting these types of conversations. The nature of "maybe sync replication represents a liability or is feasible" might be a conversation about the geography of a region and statistical likelihood of certain disasters impacting multiple sites simultaneously. Some customers cant ever afford to lose a single transaction no matter what, some just need it to be reasonably stable but incredibly fast (e.g. gaming vs banking). Will definitely be spending some time reviewing Lamport's works again. Establishing the notion of stable time between all participants is a fascinating way to solve a lot of problems in distributed systems.

brian_cunnie 5y ago

I’ve only seen one burst pipe bring down a computer (Pitney Bowes, 1995), but I’ve witnessed several S3 outages.

zthrowaway 5y ago
You should spend a week working on a infra / ops team then.
- Demiurge 5y ago
  I worked with multiple ops/infra teams, and have seen standard server rooms and advanced server rooms. They’re not supposed to have pipes leaking on the racks, that’s ridiculous. I know many stable redhat servers that didn’t need to reboot for years. I also maintain websites on hetzner servers that haven’t needed to reboot in years, under pretty high load. On the other hand, I got a hostnoc dedicated server and that thing went offline three times a month. So yeah, you have to know what you’re doing regardless of whether you’re building a server room or a k8s cluster.
  - mwcampbell 5y ago
    > I know many stable redhat servers that didn’t need to reboot for years. That's not actually something to brag about, as it implies they weren't receiving important kernel updates, particularly security updates.
    - bob1029 5y ago
      So, we do ~daily restarts on our customers' systems. We have a solid 12 hours per day to patch up a server before it needs to be in production again. That said. We communicate directly with other systems on the same premises that probably haven't been rebooted in over a decade. Building something that can run 10 years without downtime is not happenstance. It is very intentional and deliberate engineering.
    - fxtentacle 5y ago
      https://en.wikipedia.org/wiki/Kpatch https://en.wikipedia.org/wiki/Kpatch Allows you to patch the kernel without reboot.
    - Demiurge 5y ago
      As mentioned above, they were, both RedHat and Ubuntu now support kernel updates without restart.
  - gravypod 5y ago
    More modern data centers have liquid pumps available for low-cost cooling.

hnjst 5y ago

Completely aligned with my parent, day-to-day operations are often slightly more complex than making sure your server run your app properly, I'd argue that our services are increasingly dependent on others (in both directions, dependencies multiply and are more and more critical). That's also by interacting more with external entities that they bring more value. > The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system. This part was what I felt deserved a counter-point though. Consensus is indeed at the core of the issue once you want distributed fault tolerance. However, I think you'll quickly hit two things with your approach: 1-2ms of latency, I fear that it may come with highly correlated failures on the "first tier of resilience". Moreover, the "second tier" being much farther, keeping them in consensus will imply harsh trade-offs. If you use synchronous consensus protocols, you'll slow down drastically the "first tier" (assuming you want consistency), if you go for the asynchronous replication (not consensus, this matters...) then the second tier can't really intervene on leader election or failover without risking a partition on a false positive (and if you try to be conservative there your RPO will suffer). If you're into these (fundamental) issues, I'd recommend Leslie Lamport's work (i.e. https://www.college-de-france.fr/site/en-martin-abadi/seminar-2011-05-11-11h00.htm https://www.college-de-france.fr/site/en-martin-abadi/semina... or http://www.lamport.org/ http://www.lamport.org/), the paper pointing a disappointing impossibility: https://dl.acm.org/doi/abs/10.1145/3149.214121 https://dl.acm.org/doi/abs/10.1145/3149.214121 and its generalization (https://dl.acm.org/doi/abs/10.1145/167088.167119 https://dl.acm.org/doi/abs/10.1145/167088.167119).

bob1029 5y ago
I appreciate the commentary. I feel most of this effectively boils down to "it depends". Clearly, there is no one-size-fits-all solution that can make everyone happy all the time, especially considering all the constraints & variables present across the entire vertical. Where your datacenters are geographically located is usually a big first step in even starting these types of conversations. The nature of "maybe sync replication represents a liability or is feasible" might be a conversation about the geography of a region and statistical likelihood of certain disasters impacting multiple sites simultaneously. Some customers cant ever afford to lose a single transaction no matter what, some just need it to be reasonably stable but incredibly fast (e.g. gaming vs banking). Will definitely be spending some time reviewing Lamport's works again. Establishing the notion of stable time between all participants is a fascinating way to solve a lot of problems in distributed systems.