4 ms·

You're essentially describing a two-phase commit ie. commit the data to each node, and then to the whole cluster (or discard the transaction if not on every nod

You're essentially describing a two-phase commit ie. commit the data to each node, and then to the whole cluster (or discard the transaction if not on every node). This is fine for some distributed tasks, but you have to wait for the whole cluster before you can use the committed data - not fast compared to single-phase commits with eventual consistency. This is what CRDTs, Paxos and CALM are trying to solve, a single phase commit where each node can then work out the order of transactions themselves, in order to reach an identical state to the next node. In your MMO case, it looks like an ideal problem for 2-phase commits to solve :) There's also a related problem, where the volume of data you're sending is such that you can't replicate it across the whole cluster, only some of it. You then have to a) sort out the redundancy of the data, and b) the consistency of the data bearing in mind that a node may have data reliant on data from another node. Many of these issues are somewhat related to GC, distributed hash (merkle) trees, CRDTs and actors operating on data (rather than the other way round). It's a rich problem area with many coordination applications for the good solutions!

bullen 6y ago
You don't need speed when your transport is completely non-blocking async. end-to-end because your CPUs are not IO-waiting on anything. This scales way better and the time it takes to distribute over my global cluster is very little above the latency between the furthest nodes (~200ms). What user case do they have for a distributed system that is faster than 200ms for global persistent data? 200ms for a registration or purchase is not huge? You can obviously use the data before the transaction is complete anyway?! I think Java is going to have a very strong revival on the server side once the Google lawsuit is over. I cannot explain the performance I'm getting yet but I'm working on that: https://github.com/tinspin/rupy/wiki https://github.com/tinspin/rupy/wiki (Under "Why Rupy is using Java")
- lostcolony 6y ago
  If you never have a netsplit, that works fine. The whole point of CAP is not what happens when everything is up and talking to each other; the point is what happens when things start to fail (the 'P'). Let's go with two nodes (since you have no quorum, that's fine), A and B. A takes a write. Tells B about it. Now what happens? If B commits, acks to A, and -the network fails-, so that A never receives the ack (and thus says "didn't write"), you have a data split. B has committed data A has rejected as failed (and this of course becomes more and more likely the more distributed nodes that you have). How do you resolve that? Maybe you don't; it might not matter. A future write (including a retry, since the writing node knows the distributed state is 'indeterminate') might update both, and you'll never notice. Even if a read is stale, it might not be noticed, or if it is just be shrugged away as a "can't reproduce". Of course, that assumes that B is reachable before you notice an issue; if it's a long term netsplit, A can never write over to B, and depending how you handle that you either can't take a write, or you allow B to grow very stale, and yet not -know- that it's stale (so you presumably are still allowing reads from it). An eventually consistent scheme would accept and attempt to resolve that disparity; in fact, given that the data most likely to be noticed is user data, and users are likely only talking to one node at a time, you likely have an ideal case for eventual consistency.
  - bullen 6y ago
    In practice preparing for that kind of failure is a complete waste of time because you can never guarantee anything, the world is chaos and only "dead trees" (oil, coal and gas) make us believe that we are perfect. In the 4 years of running this MMO we have had less than 0.3% refund because of network problems (including antivirus blocking of real-time messages, the MMO also uses HTTP). Personally I have never seen this database fail, it's much closer to perfect than any other distributed database if you count performance per watt and total cost of operation.
    - lostcolony 6y ago
      "In practice preparing for that kind of failure is a complete waste of time because you can never guarantee anything" Oh, but you can. Hence the decades of research. And determining exactly what can be guaranteed, when, and what the tradeoffs are, is the whole point. You're using refunds as a proxy for system correctness. That may make sense for your particular business case, but it doesn't make sense for many others. And it definitely isn't sufficient for a generic technical implementation that has to be able to be evaluated for an arbitrary business case.
lostcolony 6y ago
Doesn't actually look like there is the latter "yep; commit" decision. Or any rollback or other recognition of failure.
- bullen 6y ago
  True, my system is 1-phase, N-latched.
- alexisread 6y ago
  Oops my bad, didn't look very closely at the repo. I think the gist of my text stands though, and as you've said it's not resilient to partition with multi-node updates, rollback or deletions (tombstones).

bullen 6y ago

You don't need speed when your transport is completely non-blocking async. end-to-end because your CPUs are not IO-waiting on anything. This scales way better and the time it takes to distribute over my global cluster is very little above the latency between the furthest nodes (~200ms). What user case do they have for a distributed system that is faster than 200ms for global persistent data? 200ms for a registration or purchase is not huge? You can obviously use the data before the transaction is complete anyway?! I think Java is going to have a very strong revival on the server side once the Google lawsuit is over. I cannot explain the performance I'm getting yet but I'm working on that: https://github.com/tinspin/rupy/wiki https://github.com/tinspin/rupy/wiki (Under "Why Rupy is using Java")

lostcolony 6y ago
If you never have a netsplit, that works fine. The whole point of CAP is not what happens when everything is up and talking to each other; the point is what happens when things start to fail (the 'P'). Let's go with two nodes (since you have no quorum, that's fine), A and B. A takes a write. Tells B about it. Now what happens? If B commits, acks to A, and -the network fails-, so that A never receives the ack (and thus says "didn't write"), you have a data split. B has committed data A has rejected as failed (and this of course becomes more and more likely the more distributed nodes that you have). How do you resolve that? Maybe you don't; it might not matter. A future write (including a retry, since the writing node knows the distributed state is 'indeterminate') might update both, and you'll never notice. Even if a read is stale, it might not be noticed, or if it is just be shrugged away as a "can't reproduce". Of course, that assumes that B is reachable before you notice an issue; if it's a long term netsplit, A can never write over to B, and depending how you handle that you either can't take a write, or you allow B to grow very stale, and yet not -know- that it's stale (so you presumably are still allowing reads from it). An eventually consistent scheme would accept and attempt to resolve that disparity; in fact, given that the data most likely to be noticed is user data, and users are likely only talking to one node at a time, you likely have an ideal case for eventual consistency.
- bullen 6y ago
  In practice preparing for that kind of failure is a complete waste of time because you can never guarantee anything, the world is chaos and only "dead trees" (oil, coal and gas) make us believe that we are perfect. In the 4 years of running this MMO we have had less than 0.3% refund because of network problems (including antivirus blocking of real-time messages, the MMO also uses HTTP). Personally I have never seen this database fail, it's much closer to perfect than any other distributed database if you count performance per watt and total cost of operation.
  - lostcolony 6y ago
    "In practice preparing for that kind of failure is a complete waste of time because you can never guarantee anything" Oh, but you can. Hence the decades of research. And determining exactly what can be guaranteed, when, and what the tradeoffs are, is the whole point. You're using refunds as a proxy for system correctness. That may make sense for your particular business case, but it doesn't make sense for many others. And it definitely isn't sufficient for a generic technical implementation that has to be able to be evaluated for an arbitrary business case.

lostcolony 6y ago

Doesn't actually look like there is the latter "yep; commit" decision. Or any rollback or other recognition of failure.

bullen 6y ago
True, my system is 1-phase, N-latched.
alexisread 6y ago
Oops my bad, didn't look very closely at the repo. I think the gist of my text stands though, and as you've said it's not resilient to partition with multi-node updates, rollback or deletions (tombstones).