4 ms·

The OP is correct in theory, but not in practice. If you have a lot of money and are willing and able to build a private network with enough redundancy, you ca

The OP is correct in theory, but not in practice. If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. See https://cloud.google.com/blog/products/databases/inside-cloud-spanner-and-the-cap-theorem https://cloud.google.com/blog/products/databases/inside-clou... Notice who the author is: Eric Brewer

lostcolony 5y ago
Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?)
- linuxhansl 5y ago
  You try another connection (after some timeout)? Or you start N requests in parallel and then pick the first response?
  - whateveracct 5y ago
    aren't retry policies inherently compromising on "A"? they fail and cause things to be slow when they take effect
  - lostcolony 5y ago
    "After some timeout" - exactly. During that time you effectively have a partition; you have either accepted the write, meaning you are no longer consistent, or you are blocking on the write, meaning you are no longer available. If you have multiple, let's expand out to 3 instances (not necessary, but helps illustrate some of the issues better). A tells B something 10x. A also tells C something 10x. Even assuming you don't get inconsistent responses (i.e., B has both accepted and rejected the write), what happens if B accepts it, and C rejects it? Likewise, if you have high load, you're going to have race conditions leading to lack of consistency as well (A tells B and C x = 2, meanwhile C tells A and B that x = 3). You are favoring availability > consistency.
- throwaway3699 5y ago
  Then you have downtime. If the downtime is small enough, then the trade-off can be worthwhile.
  - owenmarshall 5y ago
    You sacrificed A.
    - throwaway3699 5y ago
      So does Spanner. The point is that Spanner has so little downtime that it's not really an issue.
      - lostcolony 5y ago
        Spanner is a strongly consistent system leveraging Paxos. Yes, with sufficient redundancy you can reduce the likelihood of a partition. You still want to know what the guarantees of the system are. Spanner guarantees that if a transaction can't be agreed to by a quorum of nodes, it will be rejected. Ergo, favoring consistency. That guarantee matters, no matter how infrequent partitions are.
  - lostcolony 5y ago
    So...not available.
- toast0 5y ago
  > Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?) If you really need things to arrive, you'd be sending them on more than one connection. Of course, the difficulty with redundant fiber connections is that really the only way to determine if redundant fiber connections are in the same bundle is with a backhoe.
linuxhansl 5y ago
Not sure why you got downvoted. I came here to say the same. While it is in theory true that you cannot have a CA system, in practice if you control your H/W and data centers as well, you can build enough redundancy that a P is not more likely than (say) being hit by a meteor. So still not a CA system, but practically CA 99.9..% (add as many 9's as you have money). Edit: Spelling.
- __jem 5y ago
  99.9 isn't "A" though -- that's something like 10 hours of downtime per year. Even at 6 9s, 30 seconds is plenty of time to end up in a split brain situation!
deleted 5y ago
[deleted]
redis_mlc 5y ago
> If you have a lot of money and are willing and able to build a private network with enough redundancy Google has done this. Possibly AT&T also. > Notice who the author is: Eric Brewer Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes. As a DBA, I find CAP totally inadequate for any business purpose. Try explaining to a business owner, "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?"
- packetslave 5y ago
  Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes Teacher, academic, and vice president of infrastructure at Google. He wrote about the CAP implications of Spanner here as well: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-...
- Dylan16807 5y ago
  I think you're conflating different things there. If you built a highly redundant network, and it still partitioned, then the actual problem is that you somehow lost 80% of your lines to the internet at the same time. So your response back needs to be that your HA setup kept things running for a while but it would take actual magic to keep going once just about all the servers are unplugged simultaneously. That's not a problem with CAP, that's a problem with explaining the wrong part of the issue.
- lostcolony 5y ago
  >>> "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?" "All that money spent on HA was what allowed us to accept purchases -at all-"
lmm 5y ago
> If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. Actually I'd say this is something that's correct in theory but not in practice. In theory you can build a network so redundant that you will never experience a partition. In practice you will always have bugs in your redundancy arrangement, operational errors by sysadmins, and so on. > If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. You still have to choose whether to drop C, A, or both when P does happen. Five 9s means about an hour of downtime a year; maybe for some businesses that's small enough to be irrelevant, but by no means all.

lostcolony 5y ago

Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?)

linuxhansl 5y ago
You try another connection (after some timeout)? Or you start N requests in parallel and then pick the first response?
- whateveracct 5y ago
  aren't retry policies inherently compromising on "A"? they fail and cause things to be slow when they take effect
- lostcolony 5y ago
  "After some timeout" - exactly. During that time you effectively have a partition; you have either accepted the write, meaning you are no longer consistent, or you are blocking on the write, meaning you are no longer available. If you have multiple, let's expand out to 3 instances (not necessary, but helps illustrate some of the issues better). A tells B something 10x. A also tells C something 10x. Even assuming you don't get inconsistent responses (i.e., B has both accepted and rejected the write), what happens if B accepts it, and C rejects it? Likewise, if you have high load, you're going to have race conditions leading to lack of consistency as well (A tells B and C x = 2, meanwhile C tells A and B that x = 3). You are favoring availability > consistency.
throwaway3699 5y ago
Then you have downtime. If the downtime is small enough, then the trade-off can be worthwhile.
- owenmarshall 5y ago
  You sacrificed A.
  - throwaway3699 5y ago
    So does Spanner. The point is that Spanner has so little downtime that it's not really an issue.
    - lostcolony 5y ago
      Spanner is a strongly consistent system leveraging Paxos. Yes, with sufficient redundancy you can reduce the likelihood of a partition. You still want to know what the guarantees of the system are. Spanner guarantees that if a transaction can't be agreed to by a quorum of nodes, it will be rejected. Ergo, favoring consistency. That guarantee matters, no matter how infrequent partitions are.
- lostcolony 5y ago
  So...not available.
toast0 5y ago
> Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?) If you really need things to arrive, you'd be sending them on more than one connection. Of course, the difficulty with redundant fiber connections is that really the only way to determine if redundant fiber connections are in the same bundle is with a backhoe.

linuxhansl 5y ago

Not sure why you got downvoted. I came here to say the same. While it is in theory true that you cannot have a CA system, in practice if you control your H/W and data centers as well, you can build enough redundancy that a P is not more likely than (say) being hit by a meteor. So still not a CA system, but practically CA 99.9..% (add as many 9's as you have money). Edit: Spelling.

__jem 5y ago
99.9 isn't "A" though -- that's something like 10 hours of downtime per year. Even at 6 9s, 30 seconds is plenty of time to end up in a split brain situation!

deleted 5y ago

[deleted]

redis_mlc 5y ago

> If you have a lot of money and are willing and able to build a private network with enough redundancy Google has done this. Possibly AT&T also. > Notice who the author is: Eric Brewer Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes. As a DBA, I find CAP totally inadequate for any business purpose. Try explaining to a business owner, "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?"

packetslave 5y ago
Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes Teacher, academic, and vice president of infrastructure at Google. He wrote about the CAP implications of Spanner here as well: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-...
Dylan16807 5y ago
I think you're conflating different things there. If you built a highly redundant network, and it still partitioned, then the actual problem is that you somehow lost 80% of your lines to the internet at the same time. So your response back needs to be that your HA setup kept things running for a while but it would take actual magic to keep going once just about all the servers are unplugged simultaneously. That's not a problem with CAP, that's a problem with explaining the wrong part of the issue.
lostcolony 5y ago
>>> "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?" "All that money spent on HA was what allowed us to accept purchases -at all-"

lmm 5y ago

> If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. Actually I'd say this is something that's correct in theory but not in practice. In theory you can build a network so redundant that you will never experience a partition. In practice you will always have bugs in your redundancy arrangement, operational errors by sysadmins, and so on. > If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. You still have to choose whether to drop C, A, or both when P does happen. Five 9s means about an hour of downtime a year; maybe for some businesses that's small enough to be irrelevant, but by no means all.