4 ms·

Raft requires both a stable leader and a majority of replicas to be reachable in order to advance the consensus protocol. I'd like to be able to lose more than

Raft requires both a stable leader and a majority of replicas to be reachable in order to advance the consensus protocol. I'd like to be able to lose more than a majority of replicas and require no leaders because I don't care about consistency. I care that as long as even one node in the system is alive, it will accept writes, and those writes will eventually propagate and converge on the union set of all writes ever observed, without obliterating any ACK'd observed writes, as the system as a whole moves toward a less degraded state. I'd want topics themselves to be able to be basically "free" to create, and the write latency to remain stable over arbitrarily large datasets. Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second).

jasonwatkinspdx 11y ago
> I don't care about consistency. Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability. > Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second). Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed. Edit: Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.png http://charap.co/wp-content/uploads/2015/10/latency_over_thr...
- im_down_w_otp 11y ago
  What happens when I don't have a majority of Raft replicas alive?
  - jasonwatkinspdx 11y ago
    Then raft locks until you do again. We live in the era of on demand infrastructure. With even just 3 replicas and some reasonable automation it's extremely unlikely you'll see any unavailability. Or run 5 way across two different systems (ie 3 at google, 2 at AWS). That setup is less than $100/month to run now. You seem extremely concerned about a fault tolerant system experiencing more faults than the majority, which I think is quite easily dealt with, and utterly blithe about making inconsistency your best case. At this kind of request rate I'd honestly just shove things in postgres and call it good. There's just no reason to build out a bunch of CRDT complexity when you don't _need_ it.
    - im_down_w_otp 11y ago
      The system that's my benchmark to go up against has seen 6 seconds of downtime in 25 years. Does that give you a better perspective for my availability concerns? I also don't have "on demand" infrastructure. I have to deploy to a fixed footprint of bare metal hardware. I'm "utterly blithe" about inconsistency being my best case for a variety of reasons: 1) I know my use case extremely well. 2) I know that within that use case there are several places where availability is the #1, #2, and #3 priority because the system being unavailable for any time at all has a non-trivial direct financial impact. 3) I know within that use case where there are necessary points of consistency and how to move them out of the critical path that requires extreme availability. 4) I know exactly the problem that CRDTs solve for me, and exactly why I need them in this case, and how to avoid them with simpler immutable/idempotent write patterns where possible. 5) I have to be able to tolerate very high proportion of the infrastructure experiencing temporary and/or permanent failure because the places I have to deploy into are no where near the level of reliability of even a low-mid tier data center in the US. Janky telcos, natural disasters, malicious operators, and flimsy power delivery are the norm.
      - jasonwatkinspdx 11y ago
        And what consistency assurances does it provide? Edit: I see you edited in a longer comment after my question above. I was just curious, not looking to pick a fight. Suffice it to say we have very different perspectives on how to attack those requirements. I'm bowing out now, cheers!.
        im_down_w_otp 11y ago
        The problem is this: 1) Anything that is suggested which avoids data loss via consistency models is going to reduce my availability to unacceptable levels. 2) Anything that is suggested which keeps my availability up where it needs to be and doesn't use CRDTs, and/or a sufficiently similar semantic guarantee, is going to silently lose data due to "lost-updates" (ACK'd writes that get obliterated during conflict merges and are now gone forever). At a minimum I need sibling behavior for conflicting writes to ensure that some half-baked LWW strategy that's barely better than "pick at random" doesn't kill my data (which has very high regulatory and audit requirements around it), and then taking that a step further CRDTs, implemented correctly, provide me a nice interface for interacting with multiple version conflicts and resolving them in coherent ways without forcing me, or my other developers, to have to think up case-by-case resolution strategies and verify their correctness. There really isn't a tremendous amount of complexity to using CRDTs. That's the point of them. They're easy to use. The complexity I'd really like to avoid if possible by avoiding CRDTs is the cognitive burden my team needs to have around distributed data and all it's horrible nuances to satisfy our use case. The knowing when and why to use them, not so much the how. "At this kind of request rate I'd honestly just shove things in postgres and call it good." is very, very far away from a workable solution because my problem isn't request rate. The number of ops/sec could be hundreds or millions, and it doesn't really change my actual problem. No matter what the request rate is I could just write to a bunch of postgres instances that have a chash ring overlaid onto them... except now I have to deal with my actual problem still, which is redundancy, and local consistency in any postgres replica doesn't guarantee me much, I can't introduce a strong coordinator like a consensus protocol over the top of them, because it dumps my availability, and so I have to figure out how to safely deal with merge conflicts between the replicas... now I'm stuck figuring out how to build CRDTs on top of postgres anyway. So... what exactly did I win?

jasonwatkinspdx 11y ago

> I don't care about consistency. Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability. > Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second). Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed. Edit: Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.png http://charap.co/wp-content/uploads/2015/10/latency_over_thr...

im_down_w_otp 11y ago
What happens when I don't have a majority of Raft replicas alive?
- jasonwatkinspdx 11y ago
  Then raft locks until you do again. We live in the era of on demand infrastructure. With even just 3 replicas and some reasonable automation it's extremely unlikely you'll see any unavailability. Or run 5 way across two different systems (ie 3 at google, 2 at AWS). That setup is less than $100/month to run now. You seem extremely concerned about a fault tolerant system experiencing more faults than the majority, which I think is quite easily dealt with, and utterly blithe about making inconsistency your best case. At this kind of request rate I'd honestly just shove things in postgres and call it good. There's just no reason to build out a bunch of CRDT complexity when you don't _need_ it.
  - im_down_w_otp 11y ago
    The system that's my benchmark to go up against has seen 6 seconds of downtime in 25 years. Does that give you a better perspective for my availability concerns? I also don't have "on demand" infrastructure. I have to deploy to a fixed footprint of bare metal hardware. I'm "utterly blithe" about inconsistency being my best case for a variety of reasons: 1) I know my use case extremely well. 2) I know that within that use case there are several places where availability is the #1, #2, and #3 priority because the system being unavailable for any time at all has a non-trivial direct financial impact. 3) I know within that use case where there are necessary points of consistency and how to move them out of the critical path that requires extreme availability. 4) I know exactly the problem that CRDTs solve for me, and exactly why I need them in this case, and how to avoid them with simpler immutable/idempotent write patterns where possible. 5) I have to be able to tolerate very high proportion of the infrastructure experiencing temporary and/or permanent failure because the places I have to deploy into are no where near the level of reliability of even a low-mid tier data center in the US. Janky telcos, natural disasters, malicious operators, and flimsy power delivery are the norm.
    - jasonwatkinspdx 11y ago
      And what consistency assurances does it provide? Edit: I see you edited in a longer comment after my question above. I was just curious, not looking to pick a fight. Suffice it to say we have very different perspectives on how to attack those requirements. I'm bowing out now, cheers!.
      - im_down_w_otp 11y ago
        The problem is this: 1) Anything that is suggested which avoids data loss via consistency models is going to reduce my availability to unacceptable levels. 2) Anything that is suggested which keeps my availability up where it needs to be and doesn't use CRDTs, and/or a sufficiently similar semantic guarantee, is going to silently lose data due to "lost-updates" (ACK'd writes that get obliterated during conflict merges and are now gone forever). At a minimum I need sibling behavior for conflicting writes to ensure that some half-baked LWW strategy that's barely better than "pick at random" doesn't kill my data (which has very high regulatory and audit requirements around it), and then taking that a step further CRDTs, implemented correctly, provide me a nice interface for interacting with multiple version conflicts and resolving them in coherent ways without forcing me, or my other developers, to have to think up case-by-case resolution strategies and verify their correctness. There really isn't a tremendous amount of complexity to using CRDTs. That's the point of them. They're easy to use. The complexity I'd really like to avoid if possible by avoiding CRDTs is the cognitive burden my team needs to have around distributed data and all it's horrible nuances to satisfy our use case. The knowing when and why to use them, not so much the how. "At this kind of request rate I'd honestly just shove things in postgres and call it good." is very, very far away from a workable solution because my problem isn't request rate. The number of ops/sec could be hundreds or millions, and it doesn't really change my actual problem. No matter what the request rate is I could just write to a bunch of postgres instances that have a chash ring overlaid onto them... except now I have to deal with my actual problem still, which is redundancy, and local consistency in any postgres replica doesn't guarantee me much, I can't introduce a strong coordinator like a consensus protocol over the top of them, because it dumps my availability, and so I have to figure out how to safely deal with merge conflicts between the replicas... now I'm stuck figuring out how to build CRDTs on top of postgres anyway. So... what exactly did I win?