6 ms·

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Sure, as soon as you dec

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw.

valenterry 3y ago
They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.
- jameshart 3y ago
  Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.
  - largbae 3y ago
    "All availability rests on some assumed radius of safety", thank you for that! If this is not a named law could you name it? I have needed this statement in the past but it never came together like that.
    - anyonecancode 3y ago
      I agree it's a good point. A nuclear blast is extrinsic to the system. I think what something like CAP gets us to think about is failures/limitations that are _intrinsic_ to the system. Someone walking around the data center and unplugging all the machines, then flying to the next data center and unplugging them, etc, would also cause all of CAP to fail. Though, thinking about it more, what it also tells us is that we need to be clear about what "the system" is. There are, after all, some systems that _do_ take nuclear blasts into account, or systems that take "physical breach by a hostile actor" into account. Less dramatically, we make trade-offs all the time as to what we consider "in" system vs "out", and it's good to be conscious and explicit about them. We see this a lot in UI, especially web UI in terms of things like what browsers to support, whether we care about users' battery life or data cap limits, if we foresee ourselves running the system in a jurisdiction with different regulations around data collection and usage, etc.
  - valenterry 3y ago
    Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific part, whereas in a distributed system you can by design increase availability by adding more nodes, at least up to a certain point. Think about it: I can run the two servers I mentioned about in the same AZ. But I can also move them to a separate AZ each. For a distributed system it doesn't matter conceptually. That doesn't mean that there is no impact on the availability or performance, it just means that I don't have to change the logic of my system. For a mainframe however that's not true - and that is what makes the charm of it: because I don't have (and don't want) to care, which simplifies things a lot; at the expensive of (the option of) the availability of a distributed system.
    - filleokus 3y ago
      I've been "arguing" AGAINST mainframes on HN in the past, and after that I've seen many really cool presentations about them. One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart. So if one explodes mid instruction it will still continue fine. As for a single system, I don't think there is any single point of failure, you can swap cpu's mid operation.
      - valenterry 3y ago
        First of all, I'm not arguing against mainframes or anything! I'm just trying to clarify/explain terms and remove confusion. Just to get that out of the way. > One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart Since they are 50km apart, they are connected to each other by one or multiple cables. I assume that they are also both connected to the internet (or something else that matters) so that if one burns down the other can take over. Correct so far? If so, then please answer this questions: how does the system behave if all the cables between the two machines are severed, for an unknown time?
  - harkinian 3y ago
    No matter what, some hardware in that mainframe is making mutexes or some other kind of locking work, and that is a single point of failure. You can't have two of those be authoritative for the same thing, that's the two generals problem.
- djha-skin 3y ago
  Datacenter fires cause downtime no matter what consistency model you choose. Also, mainframes are pretty big, multi-processor machines. They have their problems, but having all those components give out all at once? I'm sure it happens, but I've never heard of it.
  - valenterry 3y ago
    > Datacenter fires cause downtime no matter what consistency model you choose. Why that? I can distribute my nodes over multiple datacenters. Netflix does that even with whole regions (to my knowledge) to avoid downtime when a region goes down.
    - djha-skin 3y ago
      In theory yes, and certainly if we're talking about Cassandra. In practice, most people do not distribute over multiple data centers. Doing so is very difficult and costly.
      - valenterry 3y ago
        Haha, I've been there. Three companies I've been with have done that - and every time we had more downtime overall than had we just run a mirrored postgres. But yeah, the theory is what I'm talking about. Not putting the theory into practice is a whole different matter. :-)
    - deleted 3y ago
      [deleted]
- deleted 3y ago
  [deleted]
- samatman 3y ago
  It's essential to rule out certain happenings when performing analysis, and acts of God are commonly among them. For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays. All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to account for cosmic rays! But you do that on top of an analysis which presumes that the hardware performs correctly. It's not a useful reality to expose to a proof assistant. To anticipate an objection: yes, you absolutely can rule out network partitions in an analysis, if that's a useful thing to do. For a packet-switched network, it isn't useful. But different network topologies exist, or at least used to, ones where once a circuit is negotiated, you can say useful things about the network without accounting for the kind of equipment failure which might break that circuit. For packet switching, you're going to have a really bad time if you don't account for partitions, because those are expected behavior, one of the properties of the system under consideration.
  - valenterry 3y ago
    That's not what this discussion is about. OP said: > as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. So we are talking in the context of distributed networks. Then: > Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw. using the words "consistency" and "availability" here is clearly refering to the same words in the sentence before. Hence we are still in the context of distributed networks. And therefore you cannot just rule this out as an "act of god". Had OP said "A mainframe has high availibility compared to my laptop and for me that is more than enough." then I wouldn't have said anything.
    - samatman 3y ago
      Your fire is just the chainsaw in the original post.
      - valenterry 3y ago
YetAnotherNick 3y ago
> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for this downtime and not thinking about much more probable ones. I have asked many who says we need three nodes and three pods for each service minimum, and no one could answer why.
- deleted 3y ago
  [deleted]
- jameshart 3y ago
  Minimum of three is generally for one of two reasons: 1) to ensure you can deliberately take one offline for maintenance and still have redundancy in case a single node goes offline 2) in some systems an odd number of nodes is needed to ensure no ties in leader elections or decision votes. Three is the smallest odd number that has any redundancy.
  - YetAnotherNick 3y ago
    1) I have taken a node in maintenance mode many times deliberately by launching second node and draining first. Its not an issue at all with just one node. For redundancy, is there ever a case where single node could go down in practice. Nodes are basically EC2 instance and they could run for years without going down. 2) 1 is odd and could win the election;)
    - jameshart 3y ago
      EC2 instances can be rebooted at any time. The underlying hardware they are on fails from time to time and they get moved. Running three nodes is a rule of thumb, not a hard rule for minimally guaranteeing availability.
      - 3y ago
throwawaymaths 3y ago
You can have a distributed system that is partially available too. Consider a GPU cluster. A network partition occurs. Both nodes can suspend jobs or refuse to accept jobs that require more than one nodes worth of compute until the partition is healed, while still independently serving jobs that require only one node, while healing all accounting details when reconciliation is possible.
harkinian 3y ago
It can be useful to say that everything is a distributed system because it actually matters in a lot of contexts, just not for CAP. There are sorts of networks within a single machine or even a single CPU, and generally "further apart" things have more latency between them. For example, task on <cpu 1, core 1> talking to another also on <1, 1> vs <1, 2> vs <2, 1> vs all the way off on the GPU... A single failure probably takes down the entire system, but not necessarily.

valenterry 3y ago

They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.

jameshart 3y ago
Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.
- largbae 3y ago
  "All availability rests on some assumed radius of safety", thank you for that! If this is not a named law could you name it? I have needed this statement in the past but it never came together like that.
  - anyonecancode 3y ago
    I agree it's a good point. A nuclear blast is extrinsic to the system. I think what something like CAP gets us to think about is failures/limitations that are _intrinsic_ to the system. Someone walking around the data center and unplugging all the machines, then flying to the next data center and unplugging them, etc, would also cause all of CAP to fail. Though, thinking about it more, what it also tells us is that we need to be clear about what "the system" is. There are, after all, some systems that _do_ take nuclear blasts into account, or systems that take "physical breach by a hostile actor" into account. Less dramatically, we make trade-offs all the time as to what we consider "in" system vs "out", and it's good to be conscious and explicit about them. We see this a lot in UI, especially web UI in terms of things like what browsers to support, whether we care about users' battery life or data cap limits, if we foresee ourselves running the system in a jurisdiction with different regulations around data collection and usage, etc.
- valenterry 3y ago
  Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific part, whereas in a distributed system you can by design increase availability by adding more nodes, at least up to a certain point. Think about it: I can run the two servers I mentioned about in the same AZ. But I can also move them to a separate AZ each. For a distributed system it doesn't matter conceptually. That doesn't mean that there is no impact on the availability or performance, it just means that I don't have to change the logic of my system. For a mainframe however that's not true - and that is what makes the charm of it: because I don't have (and don't want) to care, which simplifies things a lot; at the expensive of (the option of) the availability of a distributed system.
  - filleokus 3y ago
    I've been "arguing" AGAINST mainframes on HN in the past, and after that I've seen many really cool presentations about them. One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart. So if one explodes mid instruction it will still continue fine. As for a single system, I don't think there is any single point of failure, you can swap cpu's mid operation.
    - valenterry 3y ago
      First of all, I'm not arguing against mainframes or anything! I'm just trying to clarify/explain terms and remove confusion. Just to get that out of the way. > One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart Since they are 50km apart, they are connected to each other by one or multiple cables. I assume that they are also both connected to the internet (or something else that matters) so that if one burns down the other can take over. Correct so far? If so, then please answer this questions: how does the system behave if all the cables between the two machines are severed, for an unknown time?
- harkinian 3y ago
  No matter what, some hardware in that mainframe is making mutexes or some other kind of locking work, and that is a single point of failure. You can't have two of those be authoritative for the same thing, that's the two generals problem.
djha-skin 3y ago
Datacenter fires cause downtime no matter what consistency model you choose. Also, mainframes are pretty big, multi-processor machines. They have their problems, but having all those components give out all at once? I'm sure it happens, but I've never heard of it.
- valenterry 3y ago
  > Datacenter fires cause downtime no matter what consistency model you choose. Why that? I can distribute my nodes over multiple datacenters. Netflix does that even with whole regions (to my knowledge) to avoid downtime when a region goes down.
  - djha-skin 3y ago
    In theory yes, and certainly if we're talking about Cassandra. In practice, most people do not distribute over multiple data centers. Doing so is very difficult and costly.
    - valenterry 3y ago
      Haha, I've been there. Three companies I've been with have done that - and every time we had more downtime overall than had we just run a mirrored postgres. But yeah, the theory is what I'm talking about. Not putting the theory into practice is a whole different matter. :-)
  - deleted 3y ago
    [deleted]
deleted 3y ago
[deleted]
samatman 3y ago
It's essential to rule out certain happenings when performing analysis, and acts of God are commonly among them. For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays. All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to account for cosmic rays! But you do that on top of an analysis which presumes that the hardware performs correctly. It's not a useful reality to expose to a proof assistant. To anticipate an objection: yes, you absolutely can rule out network partitions in an analysis, if that's a useful thing to do. For a packet-switched network, it isn't useful. But different network topologies exist, or at least used to, ones where once a circuit is negotiated, you can say useful things about the network without accounting for the kind of equipment failure which might break that circuit. For packet switching, you're going to have a really bad time if you don't account for partitions, because those are expected behavior, one of the properties of the system under consideration.
- valenterry 3y ago
  That's not what this discussion is about. OP said: > as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. So we are talking in the context of distributed networks. Then: > Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw. using the words "consistency" and "availability" here is clearly refering to the same words in the sentence before. Hence we are still in the context of distributed networks. And therefore you cannot just rule this out as an "act of god". Had OP said "A mainframe has high availibility compared to my laptop and for me that is more than enough." then I wouldn't have said anything.
  - samatman 3y ago
    Your fire is just the chainsaw in the original post.
    - valenterry 3y ago

YetAnotherNick 3y ago

> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for this downtime and not thinking about much more probable ones. I have asked many who says we need three nodes and three pods for each service minimum, and no one could answer why.

deleted 3y ago
[deleted]
jameshart 3y ago
Minimum of three is generally for one of two reasons: 1) to ensure you can deliberately take one offline for maintenance and still have redundancy in case a single node goes offline 2) in some systems an odd number of nodes is needed to ensure no ties in leader elections or decision votes. Three is the smallest odd number that has any redundancy.
- YetAnotherNick 3y ago
  1) I have taken a node in maintenance mode many times deliberately by launching second node and draining first. Its not an issue at all with just one node. For redundancy, is there ever a case where single node could go down in practice. Nodes are basically EC2 instance and they could run for years without going down. 2) 1 is odd and could win the election;)
  - jameshart 3y ago
    EC2 instances can be rebooted at any time. The underlying hardware they are on fails from time to time and they get moved. Running three nodes is a rule of thumb, not a hard rule for minimally guaranteeing availability.
    - 3y ago

throwawaymaths 3y ago

You can have a distributed system that is partially available too. Consider a GPU cluster. A network partition occurs. Both nodes can suspend jobs or refuse to accept jobs that require more than one nodes worth of compute until the partition is healed, while still independently serving jobs that require only one node, while healing all accounting details when reconciliation is possible.

harkinian 3y ago

It can be useful to say that everything is a distributed system because it actually matters in a lot of contexts, just not for CAP. There are sorts of networks within a single machine or even a single CPU, and generally "further apart" things have more latency between them. For example, task on <cpu 1, core 1> talking to another also on <1, 1> vs <1, 2> vs <2, 1> vs all the way off on the GPU... A single failure probably takes down the entire system, but not necessarily.