41 ms·

Let's consign CAP to the cabinet of curiosities

135 points by nalgeon 2y ago

api 2y ago
This is just saying because the cloud system hides the implications of the theorem from you, it's not relevant. I suppose it's kinda true in the sense that how to operate a power plant is not relevant when I turn on my lights.
- hinkley 2y ago
  We pay a lot of money to Amazon to not think about the 8 Fallacies as well. You still get consequences for ignoring them, but they show up as burn rate.
- deathanatos 2y ago
  And if you actually use cloud systems long enough, you'll see violations of the "C" in CAP in the form of results that don't make any sense. So even in practice… it's good to understand, because it can and does matter.
sir-dingleberry 2y ago
The CAP theorem is irrelevant if your acceptable response time is greater than the time it takes your partitions to sync. At that point you get all 3: consistency,availability, partitioning. In my opinion it should be the CAPR theorem.
- jayzalowitz 2y ago
  and considering most cloud compute has a better backbone or internal routing, userspace should see this less. That being said, if this is truly a problem for you CRDB is basically built with this all in mind.
  - sir-dingleberry 2y ago
    Wow have heard about CRDB but just now reading what it does. That's an incredible piece of engineering.
- pessimizer 2y ago
  What's the difference between choosing an acceptable response time that is greater than the time it takes for your partitions to sync and giving up on availability? I don't think it makes sense to say that CAP doesn't apply if you don't need consistency, availability, or tolerance to partitions. CAP is entirely about the need to relax at least one of those three to shore up the others.
- anikdas 2y ago
  > The CAP theorem is also irrelevant if your acceptable response time is greater than the time it takes your partitions to sync. This is really an oversimplification. The more important metric here is the delay between write and read of the same data. Even in that case if when the system write load is unpredictable it will definitely lead to high variance in replication lag. The number of times I had to deal with a race condition for not considering replication lag factor is more than I would like to admit.
- alanwreath 2y ago
  I mean you just KNOW that while that addition may make sense, some 12 year minded person like me would just start referring to it as the CRAP theorem. And I don’t even dislike the theorem.
- adrr 2y ago
  Partition encompasses response time. If you have defined SLAs on response time and your partition times exceed that, you're not available.
  - sir-dingleberry 2y ago
    One option to increase availability though is to reduce partition times.
- hot_gril 2y ago
  You mean CLAP? Latency and availability are basically the same thing. CAP is simple.
  - sir-dingleberry 2y ago
    No I'm talking about response time 'requirements'
ivan_gammel 2y ago
The CAP theorem is quantum mechanics of software with C*A = O(1) in theory, similarly to uncertainty principle, but in many use cases this value is so small that "classical" expectations of both C and A are fine.
draw_down 2y ago
[dead]
justinsaccount 2y ago
> None of the clients need to be aware that a network partition exists (except a small number who may see their connection to the bad side drop, and be replaced by a connection to the good side). What a convenient world where the client is not affected by the network partition.
kstrauser 2y ago
So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B? A bank: No! If region A goes down, do not process updates in B until A is back up! We’d rather be down than wrong! A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic! CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior when something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.
- candiddevmike 2y ago
  How do you setup a multi-region RDS with multi-region write?
  - kstrauser 2y ago
    I think that's a thing? It's been a while since I've looked. Even if not, multi-AZ replication with failover is a standard setup and it has the same issues. Suppose you have some frontend servers and an RDS primary instance in us-west-2a and other frontend servers and an RDS replica in us-west-2b. The link between AZs goes down. For simplicity, pretend a failover doesn't happen. (It wouldn't matter if it did. All that would change here is the names of the AZs.) Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition? You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region. (I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)
senorrib 2y ago
I think every couple of months there's yet another article saying the CAP theorem is irrelevant. The problem with these is that they ignore the fact that CAP theorem isn't a guide, a framework or anything else. It's simply the formalization of a fact, and whether or not that fact is *important* (although still a fact) depends on the actual use case. Hell, it applies even to services within the same memory space, although obviously the probability of losing any of the three is orders of magnitude less than on a network. Can we please move on?
- da_chicken 2y ago
  > Can we please move on? Doubtful. If there's one thing that new developers have always insisted upon doing it's telling the networking and data store engineers that their foundational limitations are wrong. It seems that everyone has to learn that computers aren't magic through misunderstanding and experience.
  - hinkley 2y ago
    Way way back, I had a coworker who aspired to be a hacker get really mad at me because he was just sure that spinning rust was “digital” while I was asserting to him that there’s a DAC in there and that’s how data recovery services work. They crack open the drive and read the magnetic domains to determine not just what the current bytes are but potentially what the previous ones were too. They’re digital! It’s ones and zeroes! Dude, it’s a fucking magnet. A bunch of them in fact.
    - wrs 2y ago
      Yes, any real digital device is relying on a hopefully well-separated bimodal distribution of something analog.
throwaway984393 2y ago
[dead]
mordae 2y ago
You wish. > DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition Incidentally that's where CAP makes it's appearance and bites your ass. No amount of VRRP, UCARP wishful thinking can guarantee a conclusion on what partition is "correct" in presence of a network partition between load balancer nodes. Also, who determines where to point the DNS? A single point of failure VPS? Or perhaps a group of distributed machines voting? Yeah. You still need to perform the analysis. It's just that some cloud providers offer the distributed voting clusters as a service and take care of the DNS and load balancer switchover for you. And that's still not enough, because you might not want to allow stragglers write to orphan databases before the whole network fencing kicks in.
- doctorpangloss 2y ago
  I don’t know. People spend a lot of time thinking about network partitions. What’s the difference between a network partition, and a network that is very slow? You could communicate via carrier pigeon. Practicably, you can easily communicate via people in different regions, who already “know” which partition is correct. Networks are never really partitioned; and, the choice of denominator when calculating their speed means they’re never really transferring at 0 bits per second either. Eventually data will start transferring again. So a pretty simple application design can deal with all of this, and that’s the world we live in, and people deal with delays and move on with life, and they might bellyache about delays for the purpose of getting discounts from their vendors but they don’t really want to switch vendors. If you’re a bank in the highly competitive business of taking 2.95% fees (that should be 0.1%) on transactions, maybe this stuff matters. But like many things in life, like drug prices, that opportunity only exists in the US, it isn’t really relevant anywhere else in the world, and it’s certainly not a math problem or intrinsic as you’re making it sound. That’s just the mythology the Stripe and banks people have kind of labeled on top of their shit, which was Mongo at the end of the day.
  - 01HNNWZ0MV43FF 2y ago
    "A good driver sometimes misses a turn, a bad driver never misses a turn" A good engineer knows that all real-time systems are turn-based, and the network is _always_ partitioned
bjornsing 2y ago
I suspect the CAP theorem factored into the design of these cloud architectures, in such a way that it now seems irrelevant. But it probably was relevant in preventing a lot of other more complex designs.
- wmf 2y ago
  For example, having 3 AZs allows Paxos/Raft voting.
  - mrkeen 2y ago
    Paxos being CP.
xnorswap 2y ago
There's a better rebuttal(*) of CAP in Kleppmann's DDIA, under the title, "The unhelpful CAP theorem". I won't plagiarize his text, instead the chapter references his blogpost, "Please stop calling databases CP or AP": https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-... (*): rebuttal I think is the wrong word, but I couldn't think of better.
- cowboylowrez 2y ago
  Really good read, in fact all of these discussions are really great. I liked the linearizable explanation, like when Alice knows the tournament outcome but Bob doesn't. A super extension to this would underscore how important this is, and the danger of Alice knowing the outcome but Bob not knowing, just extend the website a bit and make it a sports gambling website. A system under such a parition would allow Alice to make a "sure thing" bet against Bob, so the constraint should be that when Bob's view is stale, it does not take bets. But how does Bobs view know its stale? It has to query Alices view! Lots of mind games to play!
throwaway71271 2y ago
When I design systems I just think about tiny traitor generals and their sneaky traitor messengers racing in the war, their clocks are broken, and some of them are deaf, blind or both. CAP or no CAP, chaos will reign. I think FLP (https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf) is better way to think about systems. I think CAP is not as relevant in the cloud because the complexity is so high that nobody even knows what is going on, so the just C part, regardless of the other letters, is ridiculously difficult even on a single computer. A book can be written just to explain write(2)'s surprise attacks. So you think you have guarantees whatever the designers said they have AP or CP, and yet.. the impossible will happen twice a day (and 3 times at night when its your on-call).
- SomeHacker44 2y ago
  Nice! A paper by Michael Fischer of Multiflow fame. I have one of their computers in my garage. Thanks!
- hot_gril 2y ago
  I can think of a lot of cloud systems that have C. Businesses will rely on a single machine running a big Oracle DB. Microservice architectures will be eventually consistent as a whole, but individual services will again have a single DB on a single machine. The single machine is a beastly distributed system in of itself, with multiple cores, CPUs, NUMA nodes, tiered caches, RAM, and disks. But when it comes down to two writers fighting over a row, it's going to consult some particular physical place in hardware for the lock.
killjoywashere 2y ago
The military lives in this world and will likely encourage people to continue thinking about it. Think about wearables on a submarine, as an example. Does the captain want to know his crew is fatigued, about to get sick, getting less exercise than they did on their last deployment? Yes. Can you talk to a cloud? No. Does the Admiral in Hawaii want to know those same answers about that boat, and every boat in the Group, eventually? Yes. For this situation, datacenter-aware databases are great. There are other solutions for other problems.
- leetrout 2y ago
  Yep. For those unaware, military networks deal with disruptions, disconnections, intermittent connectivity, and low-bandwidth (DDIL). https://www.usni.org/magazines/proceedings/sponsored/ddil-environment-cloud-management https://www.usni.org/magazines/proceedings/sponsored/ddil-en...
mrkeen 2y ago
> In practice, the redundant nature of connectivity and ability to use routing mechanisms to send clients to the healthy side of partitions Iow: You can have CAP as long as you can communicate across "partitions".
- crote 2y ago
  More precisely: partitions in the graph theory sense rarely (if ever) happen. In practice it looks more like a complete graph losing some edges. Let's say you have servers in DCs A and B, and clients at ISPs C and D. Normally servers at A and B can communicate, and clients at both C and D can each reach servers at A and B. If connectivity between DCs A and B goes down there is technically no partition, because connectivity between the clients and each DC is still working. The servers at one of the DCs can just say "go away, I lost my quorum", and the clients will connect to the other DC. This is the most likely scenario, and in practice the only one you're really interested in solving. If connectivity between ISP C and DC A goes down, there is no partition because they can just connect to DC B. If all connectivity for ISP C goes down, that is Not Your Problem. If all connectivity for DC A goes down, it doesn't matter because it no longer has clients writing to it, and they've all reconnected to DC B. To have a partition, you'd need to lose the link between DCs A and B, and lose the link from ISP C to DC B, and lose the link from ISP D to DC A. This leaves the clients from ISP C connected only to DC A, and the clients from ISP D connected only to DC B - while at the same time being unable to reconcile between DCs A and B. Is this theoretically possible? Yes, of course. Does this happen in practice - let alone in a close-enough split that you could reasonably be expected to serve both sides? No, not really.
  - justinsaccount 2y ago
    DCA: north america DCB: south america ISP C: north american ISP ISP D: south american ISP A major fiber link between north and south america is cut, disrupting all connectivity between the two areas. "lose the link between DCs A and B": check. "lose the link from ISP D to DC A.": check. replace north and south america with any two disparate locations.
  - hinkley 2y ago
lupire 2y ago
He's saying that you don't need Partition Tolerance because network is never actually Partitioned. This is exactly why the Internet and the US Interstate Highway system were invented in the first place. Or he's saying you don't need Consistency because your system isn't actually distributed; it's just a centralized system with hot backups. It's unclear what he's trying to say. No idea why he wrote the blog post. It doesn't increase my confidence in the engineering equality of his employer AWS
bunderbunder 2y ago
I once lost an entire Christmas vacation to fixing up the damage caused when an Elasticsearch cluster running in AWS responded poorly to a network partition event and started producing results that ruined our users' day (and business records) in a "costing millions of dollars" kind of way. It was a very old version of ES, and the specific behavior that led to the problem has been fixed for a long time now. But still, the fact that something like this can happen in a cloud deployment demonstrates that this article's advice rests on an egregiously simplistic perspective on the possible failure modes of distributed systems. In particular, the major premise that intermittent connectivity is only a problem on internetworks is just plain wrong. Hubs and switches flake out. Loose wires get jiggled. Subnetworks get congested. And if you're on the cloud, nobody even tries to pretend that they'll tell you when server and equipment maintenance is going to happen.
- hinkley 2y ago
  Twice I’ve had to take an Ethernet cable from an IT lead and throw it away. “It’s still good!” No it fucking isn’t. One of them compromised by pulling it out of the trash and cutting the ends off. When there’s maintenance going on in the server room and a machine they promised not to touch starts having intermittent networking problems, it’s probably a shitty cable getting jostled. There is an entire generation of devs now that have had physical hardware abstracted away and aren’t learning the lessons. Though my favorite stories are the cubicle neighbor who taps their foot leading to packet loss.
  - deleted 2y ago
    [deleted]
  - kstrauser 2y ago
    I watched a colleague diagnosis a flaky cable. Then he immediately pulled out a pocket knife and sliced the cable in 2. He said he didn't ever even want to be tempted to trust it again. That's been my model since then. I've never worked with a cable so expensive or irreplaceable that I'd want to take a chance on it a second time. I'm sure they exist. I haven't been around one.
motbus3 2y ago
If you don't care about costs...
- kstrauser 2y ago
  ...or physics.
pyrale 2y ago
Someone else tok ownership of the problem for you and sells you their solution : "The theoretical issue is irrelevant to me". Sure. Also, there's a long list of other things that are probably irrelevant to you. That is, until your provider fails and you need to understand the situation in order to provide a workaround. And slapping "load-balancers" everywhere on your schema is not really a solution, because load-balancers themselves are a distributed system with a state and are subject to CAP, as presented in the schema. > DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition. "Somehow, something somewhere will fix my shit hopefully". Also, as a sidenote, a few friends would angrily shake their "it's always DNS" cup reading this. edit: reading the rest of the blog and author's bio, I'm unsure whether the author is genuinely mistaken, or whether they're advertising their employer's product.
vmaurin 2y ago
Plot twist: in the article drawings, replica one and two are split by network, and it could fail. The author seems to not understand what the meaning of the P in CAP
- jvanderbot 2y ago
  "Non failing node" does not mean "Non partitioned node", simple as that. If you treat a partitioned node as "failed", then CAP does not apply. You've simply left it out cold with no read / write capability because you've designated a "quorum" as the in-group.
- hinkley 2y ago
  I missed your comment and just made the same comment with different words. He literally has a picture of a partition.
kristjansson 2y ago
Many things can be solved by the SEP Field[0] [0]: https://en.wikipedia.org/wiki/Somebody_else's_problem#Douglas_Adams'_SEP_field https://en.wikipedia.org/wiki/Somebody_else's_problem#Dougla...
ibash 2y ago
> if a quorum of replicas is available to the client, they can still get both strong consistency, and uncompromised availability. Then it’s not a partition.
tristor 2y ago
As someone who's worked extensively on distributed systems, including at a cloud provider, after reading this I think the author doesn't actually understand the CAP theorem or the two generals problem. Their conclusions are essentially utterly incorrect.
hinkley 2y ago
> The formalized CAP theorem would call this system unavailable, based on their definition of availability: Umm, no? That’s a picture of a partition. The partition is not able to make progress because the system is not partition tolerant. If it did it wouldn’t be consistent. It’s still available.
throw0101c 2y ago
See also: > In database theory, the PACELC theorem is an extension to the CAP theorem. It states that in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and loss of consistency (C). * https://en.wikipedia.org/wiki/PACELC_theorem https://en.wikipedia.org/wiki/PACELC_theorem
jumploops 2y ago
> The point of this post isn’t merely to be the ten billionth blog post on the CAP theorem. It’s to issue a challenge. A request. Please, if you’re an experienced distributed systems person who’s teaching some new folks about trade-offs in your space, don’t start with CAP. Yeah… no. Just because the cloud offers primitives that allow you to skip many of the challenges that the CAP theorem outlines, doesn’t mean it’s not a critical step to learning about and building novel distributed systems. I think the author is confusing systems practitioners with distributed systems researchers. I agree in some part, the former rarely needs to think about CAP for the majority of B2B cloud SaaS. For the latter, it seems entirely incorrect to skip CAP theorem fundamentals in one’s education. tl;dr — just because Kubernetes (et al.) make building distributed systems easier, it doesn’t mean you should avoid the CAP theorem in teaching or disregard it altogether.
jorblumesea 2y ago
CAP was never designed as an end all template you blindly apply to large scale systems. Think of it more as a mental starting pointing, that systems have these trade offs you need to consider. Each system you integrate has complex and nuanced requirements that don't neatly fall into clean buckets. As always Kleppmann has a great and deep answer for this. https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-...
KaiserPro 2y ago
I kinda see what the author is getting at, but I don't buy the argument. However, in the example with the network partition, it relies on proper monitoring to work out if the DB its attached to is currently in partition. managing reads is a piece of piss, mostly. Its when you need propagate write to the rest of the DB system, thats where stuff gets hairy. Now, most places can run from a single DB, especially as disks are fucking fast now. so CAP is never really that much of a problem. However when you go multi-region, thats when it gets interesting.
cryptonector 2y ago
> If the partition extended to the whole big internet that clients are on, this wouldn’t work. But they typically don’t. This is the key, that network partitions either keep some clients from accessing any servers, or they keep some servers from talking to each other. The former case is uninteresting because nothing can be done server-side about it. The latter is interesting and we can fix it with load balancers. This conflicts with the picture painted earlier in TFA where the unhappy client is somehow stuck with the unhappy server, but let's consider that just didactic. We can also not use load balancers but have the clients talk to all the servers they can reach, when we trust the clients to behave correctly. Some architectures do this, like Lustre, which is why I mention it. I see several comments here that seem to take TFA as saying that distributed consensus algorithms/protocols are not needed, but TFA does not say that. TFA says you can have consistency, availability, and partition tolerance because network partitions between servers typically don't extend to clients, and you can have enough servers to maintain quorum for all clients (if a quorum is not available it's as if the whole cloud is down, then it's not available to any clients). That is a very reasonable assertion, IMO.
- mrkeen 2y ago
  > TFA says you can have consistency, availability, and partition tolerance because network partitions between servers typically don't extend to clients, and you can have enough servers to maintain quorum for all clients It's not "partition tolerance" if you declare that partitions typically don't happen.
skywhopper 2y ago
Weird article. Different users have different priorities and that’s what the CAP theorem expresses. The article also pretends that there’s a magic “load balancer” in the cloud that always works and also knows which segment of a partitioned network is the “correct” one (one of the points of CAP is that there’s not necessarily a “correct” side), and that no users will ever be on the “wrong” side of the partition. And not only that but all replicas see the exact same network partition. None of this is reality. But the gist, I guess, is that for most applications it’s not actually that important, and that’s probably true. But when it is important, “the cloud” is not going to save you.
rdtsc 2y ago
> The CAP Theorem is Irrelevant Just sprinkle the magic "cloud" powder on your system and ignore all the theory. https://ferd.ca/beating-the-cap-theorem-checklist.html https://ferd.ca/beating-the-cap-theorem-checklist.html Let's see, let's pick some checkboxes. (x) you pushed the actual problem to another layer of the system (x) you're actually building an AP system
- hot_gril 2y ago
  You're building a ??? system. Either CP or AP depending on what random values got put into some config.
  - deleted 2y ago
    [deleted]
- Spivak 2y ago
  Specifically they pushed the problem to the load-balancer so they get to check this one too. (x) your solution requires a central authority that cannot be unavailable I'm amazed the author doesn't consider that the load-balancers in this situation are effectively acting as non-voting witnesses and that the whole system hinges on the lbs being able to correctly determine the healthy nodes. In a single-master setup the network partition that cuts the line between the lbs and the current master (but nothing else) is a fun one. Everything would be fine if you could promote a new master but who's gonna tell em'?
  - rdtsc 2y ago
    That's a good point. It describes the situation better than my checklist.
hot_gril 2y ago
Every time someone tries to deprecate the nice and simple CAP theorem, it grows stronger. It's an unstoppable freight train at this point, like the concept of relational DBs after the NoSQL fad.
kwillets 2y ago
This seems like what I've noticed on MPP systems (a little before cloud): data replicas give a lot more availability than the number of partition events would suggest. I likely need to read the paper linked, but it's common to have an MPP database lose a node but maintain data availability. CAP applies at various levels, but the notion of availability differs: 1. all nodes available 2. all data available Redundancy can make #2 a lot more common than #1.
hot_gril 2y ago
The only concrete solution the article proposes that I can think of: Spanner uses quorum to maintain availability and consistency. Your "master" is TrueTime, which is considered reliable enough. You have replicated app backends. If this isn't too generous, let's also say the cloud handles load balancing well enough. CAP isn't violated, but you might say the user no longer worries about it. Most databases don't work like Spanner, and Spanner has its downsides, two of them being cost and performance. So most of the time, you're using a traditional DB with maybe a RW replica, which will sacrifice significant consistency or availability depending on whether you choose sync or async mode. And you're back to worrying about CAP.
- hot_gril 2y ago
  Ok, you could also have multiple synchronous Postgres standbies in a quorum setup so that it's tolerable to lose one. Supposedly Amazon Aurora does that. Comes with costs and downsides too.
PaulHoule 2y ago
So glad to see that the CAP "theorem" is being recognized as a harmful selfish meme like Fielding's REST paper with a deadly seductive power against the overly pedantic.
thayne 2y ago
This only addresses one kind of partition. What if your servers can't talk to each other, but clients can? What if clients can't connect to any of your servers? What if there are multiple partitons, and none of them have a quorum? Also, changing the routing isn't instantaneous, so you will have some period of unavailability between when the partition happens, and when the client is redirected to the partition with the quorum.
rubiquity 2y ago
The point trying to be made is that with nimble infrastructure the A in CAP can be designed around to such a small amount you may as well be a CP system unless you have a really good reason to go after that 0.005% of availability. Not being CP means sacrificing the wonderful benefits that being consistent (linearizability, sequential consistency, strict serializibility) make possible. It's hard to disagree with that sentiment, and is likely why the Local First ideology is centered on data ownership rather than that extra 0.0005 ounces of availability. Once availability is no longer the center of attention the design space can be focused on durability or latency: how many copies to read/write before acking. Unfortunately the point is lost because of the usage of the word "cloud", a somewhat contrived example of solving problems by reconfiguring load balancers (in the real world certain outages might not let you reconfigure!), and missing empathy that you can't tell people not to care about how the semantics that thinking about, or not thinking about, availability imposes on the correctness of their applications. As for the usage of the word cloud: I don't know when a set of machines becomes a cloud. Is it the APIs for management? Or when you have two or more implementations of consensus running on the set of machines?
mcbrit 2y ago
All models are wrong, some are useful. CAP is probably at least as useful as Newtonian mechanics WHEN you are explaining why you just did a bunch of… extra stuff. I would like to violate CAP, please. I would like to be nearish to c, please. Here is my passport. I have done the work.
remram 2y ago
This article assumes P(artitions) don't happen, and then concludes you can have both C and A. Congrats, that's the CAP theorem.
fractalic 2y ago
Hmm this article seems misleading. I suppose it's trying to make the point that application designers usually don't need to think too hard about it, because it's already being addressed by a quorum consensus protocol implemented by someone else. This is a bit of a tautology though; the author seems to be saying 'assume you have a solution to CAP theorem -- now isn't it silly to worry about CAP theorem?'. One of the fundamental assumptions of CAP theorem is that you can't tell whether or not you have a partition. If you have an oracle that can instantaneously tell you the state of every subsystem, then yeah, CAP is pointless. But if one of your DBs is connected, reporting itself as alive, and throwing all its writes into /dev/null, you won't be able to route traffic to a quorum of healthy instances because it's not possible to be certain that they're all healthy. This is what CAP theorem is about: managing data in a distributed system where the status of any given system is fundamentally unknowable because of the Two Generals' Problem (https://en.wikipedia.org/wiki/Two_Generals'_Problem) https://en.wikipedia.org/wiki/Two_Generals'_Problem) In many cases in Cloud though, we can skip that technical stuff and design systems as if we really _did_ have an oracle that could instantaneously and perfectly tell us the state of the system, and things will typically work fine.
linuxhansl 2y ago
So basically this is saying that the CAP theorem is irrelevant because a partition is not really have a partition (since the load balancer still can reach everybody). Hmm... I agree that in modern data centers the CAP theorem is essentially irrelevant for intra-DC services, due the uptime and redundancy of networking H/W (making a partition less likely than other systemic failures). Across DCs I'll claim it is still absolutely relevant.

135 points by nalgeon 2y ago

api 2y ago

This is just saying because the cloud system hides the implications of the theorem from you, it's not relevant. I suppose it's kinda true in the sense that how to operate a power plant is not relevant when I turn on my lights.

hinkley 2y ago
We pay a lot of money to Amazon to not think about the 8 Fallacies as well. You still get consequences for ignoring them, but they show up as burn rate.
deathanatos 2y ago
And if you actually use cloud systems long enough, you'll see violations of the "C" in CAP in the form of results that don't make any sense. So even in practice… it's good to understand, because it can and does matter.

sir-dingleberry 2y ago

The CAP theorem is irrelevant if your acceptable response time is greater than the time it takes your partitions to sync. At that point you get all 3: consistency,availability, partitioning. In my opinion it should be the CAPR theorem.

jayzalowitz 2y ago
and considering most cloud compute has a better backbone or internal routing, userspace should see this less. That being said, if this is truly a problem for you CRDB is basically built with this all in mind.
- sir-dingleberry 2y ago
  Wow have heard about CRDB but just now reading what it does. That's an incredible piece of engineering.
pessimizer 2y ago
What's the difference between choosing an acceptable response time that is greater than the time it takes for your partitions to sync and giving up on availability? I don't think it makes sense to say that CAP doesn't apply if you don't need consistency, availability, or tolerance to partitions. CAP is entirely about the need to relax at least one of those three to shore up the others.
anikdas 2y ago
> The CAP theorem is also irrelevant if your acceptable response time is greater than the time it takes your partitions to sync. This is really an oversimplification. The more important metric here is the delay between write and read of the same data. Even in that case if when the system write load is unpredictable it will definitely lead to high variance in replication lag. The number of times I had to deal with a race condition for not considering replication lag factor is more than I would like to admit.
alanwreath 2y ago
I mean you just KNOW that while that addition may make sense, some 12 year minded person like me would just start referring to it as the CRAP theorem. And I don’t even dislike the theorem.
adrr 2y ago
Partition encompasses response time. If you have defined SLAs on response time and your partition times exceed that, you're not available.
- sir-dingleberry 2y ago
  One option to increase availability though is to reduce partition times.
hot_gril 2y ago
You mean CLAP? Latency and availability are basically the same thing. CAP is simple.
- sir-dingleberry 2y ago
  No I'm talking about response time 'requirements'

ivan_gammel 2y ago

The CAP theorem is quantum mechanics of software with C*A = O(1) in theory, similarly to uncertainty principle, but in many use cases this value is so small that "classical" expectations of both C and A are fine.

draw_down 2y ago

[dead]

justinsaccount 2y ago

> None of the clients need to be aware that a network partition exists (except a small number who may see their connection to the bad side drop, and be replaced by a connection to the good side). What a convenient world where the client is not affected by the network partition.

kstrauser 2y ago

So you’re setting up a multi-region RDS. If region A goes down, do you continue to accept writes to region B? A bank: No! If region A goes down, do not process updates in B until A is back up! We’d rather be down than wrong! A web forum: Yes! We can reconcile later when A comes back up. Until then keep serving traffic! CAP theorem doesn’t let you treat the cloud as a magic infinite availability box. You still have to design your system to pick the appropriate behavior when something breaks. No one without deep insight into your business needs can decide for you, either. You’re on the hook for choosing.

candiddevmike 2y ago
How do you setup a multi-region RDS with multi-region write?
- kstrauser 2y ago
  I think that's a thing? It's been a while since I've looked. Even if not, multi-AZ replication with failover is a standard setup and it has the same issues. Suppose you have some frontend servers and an RDS primary instance in us-west-2a and other frontend servers and an RDS replica in us-west-2b. The link between AZs goes down. For simplicity, pretend a failover doesn't happen. (It wouldn't matter if it did. All that would change here is the names of the AZs.) Do you accept writes in us-west-2a knowing the ones in 2b can't see them? Do you serve reads in 2b knowing they might be showing old information? Or do you shut down 2b altogether and limp along at half capacity in only 2a? What if the problem is that 2a becomes inaccessible to the Internet so that you can no longer use a load balancer to route requests to it? What if the writes in 2a hadn't fully replicated to 2b before the partition? You can probably answer those for any given business scenario, but the point is that you have to decide them and you can't outsource it to RDS. Some use cases prioritize availability above all else, using eventual consistency to work out the details once connectivity's restored. Others demand consistency above all else, and would rather be down than risk giving out wrong answers. No cloud host can decide what's write for you. The CAP theorem is extremely freaking relevant to anyone using more than one AZ or one region. (I know you weren't claiming otherwise, just taking a chance to say why cross-AZ still has the same issues as cross-region, as I have heard well meaning people say they were somehow different. AWS does a great job of keeping things running well to the point that it's news when they don't. Things still happen though. Same for Azure, GCP, and any other cloud offering. However flawless their design and execution, if a volcano erupts in the wrong place, there's gonna be a partition.)

senorrib 2y ago

I think every couple of months there's yet another article saying the CAP theorem is irrelevant. The problem with these is that they ignore the fact that CAP theorem isn't a guide, a framework or anything else. It's simply the formalization of a fact, and whether or not that fact is *important* (although still a fact) depends on the actual use case. Hell, it applies even to services within the same memory space, although obviously the probability of losing any of the three is orders of magnitude less than on a network. Can we please move on?

da_chicken 2y ago
> Can we please move on? Doubtful. If there's one thing that new developers have always insisted upon doing it's telling the networking and data store engineers that their foundational limitations are wrong. It seems that everyone has to learn that computers aren't magic through misunderstanding and experience.
- hinkley 2y ago
  Way way back, I had a coworker who aspired to be a hacker get really mad at me because he was just sure that spinning rust was “digital” while I was asserting to him that there’s a DAC in there and that’s how data recovery services work. They crack open the drive and read the magnetic domains to determine not just what the current bytes are but potentially what the previous ones were too. They’re digital! It’s ones and zeroes! Dude, it’s a fucking magnet. A bunch of them in fact.
  - wrs 2y ago
    Yes, any real digital device is relying on a hopefully well-separated bimodal distribution of something analog.

throwaway984393 2y ago

[dead]

mordae 2y ago

You wish. > DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition Incidentally that's where CAP makes it's appearance and bites your ass. No amount of VRRP, UCARP wishful thinking can guarantee a conclusion on what partition is "correct" in presence of a network partition between load balancer nodes. Also, who determines where to point the DNS? A single point of failure VPS? Or perhaps a group of distributed machines voting? Yeah. You still need to perform the analysis. It's just that some cloud providers offer the distributed voting clusters as a service and take care of the DNS and load balancer switchover for you. And that's still not enough, because you might not want to allow stragglers write to orphan databases before the whole network fencing kicks in.

doctorpangloss 2y ago
I don’t know. People spend a lot of time thinking about network partitions. What’s the difference between a network partition, and a network that is very slow? You could communicate via carrier pigeon. Practicably, you can easily communicate via people in different regions, who already “know” which partition is correct. Networks are never really partitioned; and, the choice of denominator when calculating their speed means they’re never really transferring at 0 bits per second either. Eventually data will start transferring again. So a pretty simple application design can deal with all of this, and that’s the world we live in, and people deal with delays and move on with life, and they might bellyache about delays for the purpose of getting discounts from their vendors but they don’t really want to switch vendors. If you’re a bank in the highly competitive business of taking 2.95% fees (that should be 0.1%) on transactions, maybe this stuff matters. But like many things in life, like drug prices, that opportunity only exists in the US, it isn’t really relevant anywhere else in the world, and it’s certainly not a math problem or intrinsic as you’re making it sound. That’s just the mythology the Stripe and banks people have kind of labeled on top of their shit, which was Mongo at the end of the day.
- 01HNNWZ0MV43FF 2y ago
  "A good driver sometimes misses a turn, a bad driver never misses a turn" A good engineer knows that all real-time systems are turn-based, and the network is _always_ partitioned

bjornsing 2y ago

I suspect the CAP theorem factored into the design of these cloud architectures, in such a way that it now seems irrelevant. But it probably was relevant in preventing a lot of other more complex designs.

wmf 2y ago
For example, having 3 AZs allows Paxos/Raft voting.
- mrkeen 2y ago
  Paxos being CP.

xnorswap 2y ago

There's a better rebuttal(*) of CAP in Kleppmann's DDIA, under the title, "The unhelpful CAP theorem". I won't plagiarize his text, instead the chapter references his blogpost, "Please stop calling databases CP or AP": https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-... (*): rebuttal I think is the wrong word, but I couldn't think of better.

cowboylowrez 2y ago
Really good read, in fact all of these discussions are really great. I liked the linearizable explanation, like when Alice knows the tournament outcome but Bob doesn't. A super extension to this would underscore how important this is, and the danger of Alice knowing the outcome but Bob not knowing, just extend the website a bit and make it a sports gambling website. A system under such a parition would allow Alice to make a "sure thing" bet against Bob, so the constraint should be that when Bob's view is stale, it does not take bets. But how does Bobs view know its stale? It has to query Alices view! Lots of mind games to play!

throwaway71271 2y ago

When I design systems I just think about tiny traitor generals and their sneaky traitor messengers racing in the war, their clocks are broken, and some of them are deaf, blind or both. CAP or no CAP, chaos will reign. I think FLP (https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf) is better way to think about systems. I think CAP is not as relevant in the cloud because the complexity is so high that nobody even knows what is going on, so the just C part, regardless of the other letters, is ridiculously difficult even on a single computer. A book can be written just to explain write(2)'s surprise attacks. So you think you have guarantees whatever the designers said they have AP or CP, and yet.. the impossible will happen twice a day (and 3 times at night when its your on-call).

SomeHacker44 2y ago
Nice! A paper by Michael Fischer of Multiflow fame. I have one of their computers in my garage. Thanks!
hot_gril 2y ago
I can think of a lot of cloud systems that have C. Businesses will rely on a single machine running a big Oracle DB. Microservice architectures will be eventually consistent as a whole, but individual services will again have a single DB on a single machine. The single machine is a beastly distributed system in of itself, with multiple cores, CPUs, NUMA nodes, tiered caches, RAM, and disks. But when it comes down to two writers fighting over a row, it's going to consult some particular physical place in hardware for the lock.

killjoywashere 2y ago

The military lives in this world and will likely encourage people to continue thinking about it. Think about wearables on a submarine, as an example. Does the captain want to know his crew is fatigued, about to get sick, getting less exercise than they did on their last deployment? Yes. Can you talk to a cloud? No. Does the Admiral in Hawaii want to know those same answers about that boat, and every boat in the Group, eventually? Yes. For this situation, datacenter-aware databases are great. There are other solutions for other problems.

leetrout 2y ago
Yep. For those unaware, military networks deal with disruptions, disconnections, intermittent connectivity, and low-bandwidth (DDIL). https://www.usni.org/magazines/proceedings/sponsored/ddil-environment-cloud-management https://www.usni.org/magazines/proceedings/sponsored/ddil-en...

mrkeen 2y ago

> In practice, the redundant nature of connectivity and ability to use routing mechanisms to send clients to the healthy side of partitions Iow: You can have CAP as long as you can communicate across "partitions".

crote 2y ago
More precisely: partitions in the graph theory sense rarely (if ever) happen. In practice it looks more like a complete graph losing some edges. Let's say you have servers in DCs A and B, and clients at ISPs C and D. Normally servers at A and B can communicate, and clients at both C and D can each reach servers at A and B. If connectivity between DCs A and B goes down there is technically no partition, because connectivity between the clients and each DC is still working. The servers at one of the DCs can just say "go away, I lost my quorum", and the clients will connect to the other DC. This is the most likely scenario, and in practice the only one you're really interested in solving. If connectivity between ISP C and DC A goes down, there is no partition because they can just connect to DC B. If all connectivity for ISP C goes down, that is Not Your Problem. If all connectivity for DC A goes down, it doesn't matter because it no longer has clients writing to it, and they've all reconnected to DC B. To have a partition, you'd need to lose the link between DCs A and B, and lose the link from ISP C to DC B, and lose the link from ISP D to DC A. This leaves the clients from ISP C connected only to DC A, and the clients from ISP D connected only to DC B - while at the same time being unable to reconcile between DCs A and B. Is this theoretically possible? Yes, of course. Does this happen in practice - let alone in a close-enough split that you could reasonably be expected to serve both sides? No, not really.
- justinsaccount 2y ago
  DCA: north america DCB: south america ISP C: north american ISP ISP D: south american ISP A major fiber link between north and south america is cut, disrupting all connectivity between the two areas. "lose the link between DCs A and B": check. "lose the link from ISP D to DC A.": check. replace north and south america with any two disparate locations.
- hinkley 2y ago

lupire 2y ago

He's saying that you don't need Partition Tolerance because network is never actually Partitioned. This is exactly why the Internet and the US Interstate Highway system were invented in the first place. Or he's saying you don't need Consistency because your system isn't actually distributed; it's just a centralized system with hot backups. It's unclear what he's trying to say. No idea why he wrote the blog post. It doesn't increase my confidence in the engineering equality of his employer AWS

bunderbunder 2y ago

I once lost an entire Christmas vacation to fixing up the damage caused when an Elasticsearch cluster running in AWS responded poorly to a network partition event and started producing results that ruined our users' day (and business records) in a "costing millions of dollars" kind of way. It was a very old version of ES, and the specific behavior that led to the problem has been fixed for a long time now. But still, the fact that something like this can happen in a cloud deployment demonstrates that this article's advice rests on an egregiously simplistic perspective on the possible failure modes of distributed systems. In particular, the major premise that intermittent connectivity is only a problem on internetworks is just plain wrong. Hubs and switches flake out. Loose wires get jiggled. Subnetworks get congested. And if you're on the cloud, nobody even tries to pretend that they'll tell you when server and equipment maintenance is going to happen.

hinkley 2y ago
Twice I’ve had to take an Ethernet cable from an IT lead and throw it away. “It’s still good!” No it fucking isn’t. One of them compromised by pulling it out of the trash and cutting the ends off. When there’s maintenance going on in the server room and a machine they promised not to touch starts having intermittent networking problems, it’s probably a shitty cable getting jostled. There is an entire generation of devs now that have had physical hardware abstracted away and aren’t learning the lessons. Though my favorite stories are the cubicle neighbor who taps their foot leading to packet loss.
- deleted 2y ago
  [deleted]
- kstrauser 2y ago
  I watched a colleague diagnosis a flaky cable. Then he immediately pulled out a pocket knife and sliced the cable in 2. He said he didn't ever even want to be tempted to trust it again. That's been my model since then. I've never worked with a cable so expensive or irreplaceable that I'd want to take a chance on it a second time. I'm sure they exist. I haven't been around one.

motbus3 2y ago

If you don't care about costs...

kstrauser 2y ago
...or physics.

pyrale 2y ago

Someone else tok ownership of the problem for you and sells you their solution : "The theoretical issue is irrelevant to me". Sure. Also, there's a long list of other things that are probably irrelevant to you. That is, until your provider fails and you need to understand the situation in order to provide a workaround. And slapping "load-balancers" everywhere on your schema is not really a solution, because load-balancers themselves are a distributed system with a state and are subject to CAP, as presented in the schema. > DNS, multi-cast, or some other mechanism directs them towards a healthy load balancer on the healthy side of the partition. "Somehow, something somewhere will fix my shit hopefully". Also, as a sidenote, a few friends would angrily shake their "it's always DNS" cup reading this. edit: reading the rest of the blog and author's bio, I'm unsure whether the author is genuinely mistaken, or whether they're advertising their employer's product.

vmaurin 2y ago

Plot twist: in the article drawings, replica one and two are split by network, and it could fail. The author seems to not understand what the meaning of the P in CAP

jvanderbot 2y ago
"Non failing node" does not mean "Non partitioned node", simple as that. If you treat a partitioned node as "failed", then CAP does not apply. You've simply left it out cold with no read / write capability because you've designated a "quorum" as the in-group.
hinkley 2y ago
I missed your comment and just made the same comment with different words. He literally has a picture of a partition.

kristjansson 2y ago

Many things can be solved by the SEP Field[0] [0]: https://en.wikipedia.org/wiki/Somebody_else's_problem#Douglas_Adams'_SEP_field https://en.wikipedia.org/wiki/Somebody_else's_problem#Dougla...

ibash 2y ago

> if a quorum of replicas is available to the client, they can still get both strong consistency, and uncompromised availability. Then it’s not a partition.

tristor 2y ago

As someone who's worked extensively on distributed systems, including at a cloud provider, after reading this I think the author doesn't actually understand the CAP theorem or the two generals problem. Their conclusions are essentially utterly incorrect.

hinkley 2y ago

> The formalized CAP theorem would call this system unavailable, based on their definition of availability: Umm, no? That’s a picture of a partition. The partition is not able to make progress because the system is not partition tolerant. If it did it wouldn’t be consistent. It’s still available.

throw0101c 2y ago

See also: > In database theory, the PACELC theorem is an extension to the CAP theorem. It states that in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and loss of consistency (C). * https://en.wikipedia.org/wiki/PACELC_theorem https://en.wikipedia.org/wiki/PACELC_theorem

jumploops 2y ago

> The point of this post isn’t merely to be the ten billionth blog post on the CAP theorem. It’s to issue a challenge. A request. Please, if you’re an experienced distributed systems person who’s teaching some new folks about trade-offs in your space, don’t start with CAP. Yeah… no. Just because the cloud offers primitives that allow you to skip many of the challenges that the CAP theorem outlines, doesn’t mean it’s not a critical step to learning about and building novel distributed systems. I think the author is confusing systems practitioners with distributed systems researchers. I agree in some part, the former rarely needs to think about CAP for the majority of B2B cloud SaaS. For the latter, it seems entirely incorrect to skip CAP theorem fundamentals in one’s education. tl;dr — just because Kubernetes (et al.) make building distributed systems easier, it doesn’t mean you should avoid the CAP theorem in teaching or disregard it altogether.

jorblumesea 2y ago

CAP was never designed as an end all template you blindly apply to large scale systems. Think of it more as a mental starting pointing, that systems have these trade offs you need to consider. Each system you integrate has complex and nuanced requirements that don't neatly fall into clean buckets. As always Kleppmann has a great and deep answer for this. https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-...

KaiserPro 2y ago

I kinda see what the author is getting at, but I don't buy the argument. However, in the example with the network partition, it relies on proper monitoring to work out if the DB its attached to is currently in partition. managing reads is a piece of piss, mostly. Its when you need propagate write to the rest of the DB system, thats where stuff gets hairy. Now, most places can run from a single DB, especially as disks are fucking fast now. so CAP is never really that much of a problem. However when you go multi-region, thats when it gets interesting.

cryptonector 2y ago

> If the partition extended to the whole big internet that clients are on, this wouldn’t work. But they typically don’t. This is the key, that network partitions either keep some clients from accessing any servers, or they keep some servers from talking to each other. The former case is uninteresting because nothing can be done server-side about it. The latter is interesting and we can fix it with load balancers. This conflicts with the picture painted earlier in TFA where the unhappy client is somehow stuck with the unhappy server, but let's consider that just didactic. We can also not use load balancers but have the clients talk to all the servers they can reach, when we trust the clients to behave correctly. Some architectures do this, like Lustre, which is why I mention it. I see several comments here that seem to take TFA as saying that distributed consensus algorithms/protocols are not needed, but TFA does not say that. TFA says you can have consistency, availability, and partition tolerance because network partitions between servers typically don't extend to clients, and you can have enough servers to maintain quorum for all clients (if a quorum is not available it's as if the whole cloud is down, then it's not available to any clients). That is a very reasonable assertion, IMO.

mrkeen 2y ago
> TFA says you can have consistency, availability, and partition tolerance because network partitions between servers typically don't extend to clients, and you can have enough servers to maintain quorum for all clients It's not "partition tolerance" if you declare that partitions typically don't happen.

skywhopper 2y ago

Weird article. Different users have different priorities and that’s what the CAP theorem expresses. The article also pretends that there’s a magic “load balancer” in the cloud that always works and also knows which segment of a partitioned network is the “correct” one (one of the points of CAP is that there’s not necessarily a “correct” side), and that no users will ever be on the “wrong” side of the partition. And not only that but all replicas see the exact same network partition. None of this is reality. But the gist, I guess, is that for most applications it’s not actually that important, and that’s probably true. But when it is important, “the cloud” is not going to save you.

rdtsc 2y ago

> The CAP Theorem is Irrelevant Just sprinkle the magic "cloud" powder on your system and ignore all the theory. https://ferd.ca/beating-the-cap-theorem-checklist.html https://ferd.ca/beating-the-cap-theorem-checklist.html Let's see, let's pick some checkboxes. (x) you pushed the actual problem to another layer of the system (x) you're actually building an AP system

hot_gril 2y ago
You're building a ??? system. Either CP or AP depending on what random values got put into some config.
- deleted 2y ago
  [deleted]
Spivak 2y ago
Specifically they pushed the problem to the load-balancer so they get to check this one too. (x) your solution requires a central authority that cannot be unavailable I'm amazed the author doesn't consider that the load-balancers in this situation are effectively acting as non-voting witnesses and that the whole system hinges on the lbs being able to correctly determine the healthy nodes. In a single-master setup the network partition that cuts the line between the lbs and the current master (but nothing else) is a fun one. Everything would be fine if you could promote a new master but who's gonna tell em'?
- rdtsc 2y ago
  That's a good point. It describes the situation better than my checklist.

hot_gril 2y ago

Every time someone tries to deprecate the nice and simple CAP theorem, it grows stronger. It's an unstoppable freight train at this point, like the concept of relational DBs after the NoSQL fad.

kwillets 2y ago

This seems like what I've noticed on MPP systems (a little before cloud): data replicas give a lot more availability than the number of partition events would suggest. I likely need to read the paper linked, but it's common to have an MPP database lose a node but maintain data availability. CAP applies at various levels, but the notion of availability differs: 1. all nodes available 2. all data available Redundancy can make #2 a lot more common than #1.

hot_gril 2y ago

The only concrete solution the article proposes that I can think of: Spanner uses quorum to maintain availability and consistency. Your "master" is TrueTime, which is considered reliable enough. You have replicated app backends. If this isn't too generous, let's also say the cloud handles load balancing well enough. CAP isn't violated, but you might say the user no longer worries about it. Most databases don't work like Spanner, and Spanner has its downsides, two of them being cost and performance. So most of the time, you're using a traditional DB with maybe a RW replica, which will sacrifice significant consistency or availability depending on whether you choose sync or async mode. And you're back to worrying about CAP.

hot_gril 2y ago
Ok, you could also have multiple synchronous Postgres standbies in a quorum setup so that it's tolerable to lose one. Supposedly Amazon Aurora does that. Comes with costs and downsides too.

PaulHoule 2y ago

So glad to see that the CAP "theorem" is being recognized as a harmful selfish meme like Fielding's REST paper with a deadly seductive power against the overly pedantic.

thayne 2y ago

This only addresses one kind of partition. What if your servers can't talk to each other, but clients can? What if clients can't connect to any of your servers? What if there are multiple partitons, and none of them have a quorum? Also, changing the routing isn't instantaneous, so you will have some period of unavailability between when the partition happens, and when the client is redirected to the partition with the quorum.

rubiquity 2y ago

The point trying to be made is that with nimble infrastructure the A in CAP can be designed around to such a small amount you may as well be a CP system unless you have a really good reason to go after that 0.005% of availability. Not being CP means sacrificing the wonderful benefits that being consistent (linearizability, sequential consistency, strict serializibility) make possible. It's hard to disagree with that sentiment, and is likely why the Local First ideology is centered on data ownership rather than that extra 0.0005 ounces of availability. Once availability is no longer the center of attention the design space can be focused on durability or latency: how many copies to read/write before acking. Unfortunately the point is lost because of the usage of the word "cloud", a somewhat contrived example of solving problems by reconfiguring load balancers (in the real world certain outages might not let you reconfigure!), and missing empathy that you can't tell people not to care about how the semantics that thinking about, or not thinking about, availability imposes on the correctness of their applications. As for the usage of the word cloud: I don't know when a set of machines becomes a cloud. Is it the APIs for management? Or when you have two or more implementations of consensus running on the set of machines?

mcbrit 2y ago

All models are wrong, some are useful. CAP is probably at least as useful as Newtonian mechanics WHEN you are explaining why you just did a bunch of… extra stuff. I would like to violate CAP, please. I would like to be nearish to c, please. Here is my passport. I have done the work.

remram 2y ago

This article assumes P(artitions) don't happen, and then concludes you can have both C and A. Congrats, that's the CAP theorem.

fractalic 2y ago

Hmm this article seems misleading. I suppose it's trying to make the point that application designers usually don't need to think too hard about it, because it's already being addressed by a quorum consensus protocol implemented by someone else. This is a bit of a tautology though; the author seems to be saying 'assume you have a solution to CAP theorem -- now isn't it silly to worry about CAP theorem?'. One of the fundamental assumptions of CAP theorem is that you can't tell whether or not you have a partition. If you have an oracle that can instantaneously tell you the state of every subsystem, then yeah, CAP is pointless. But if one of your DBs is connected, reporting itself as alive, and throwing all its writes into /dev/null, you won't be able to route traffic to a quorum of healthy instances because it's not possible to be certain that they're all healthy. This is what CAP theorem is about: managing data in a distributed system where the status of any given system is fundamentally unknowable because of the Two Generals' Problem (https://en.wikipedia.org/wiki/Two_Generals'_Problem) https://en.wikipedia.org/wiki/Two_Generals'_Problem) In many cases in Cloud though, we can skip that technical stuff and design systems as if we really _did_ have an oracle that could instantaneously and perfectly tell us the state of the system, and things will typically work fine.

linuxhansl 2y ago

So basically this is saying that the CAP theorem is irrelevant because a partition is not really have a partition (since the load balancer still can reach everybody). Hmm... I agree that in modern data centers the CAP theorem is essentially irrelevant for intra-DC services, due the uptime and redundancy of networking H/W (making a partition less likely than other systemic failures). Across DCs I'll claim it is still absolutely relevant.