3 ms·

It's a bit of a middle ground. Yes, the replication is synchronous, which impacts availability. However, the master can remove a failed replica from the chain f

by snewman 15y ago

It's a bit of a middle ground. Yes, the replication is synchronous, which impacts availability. However, the master can remove a failed replica from the chain fairly quickly. In principle, with proper tuning, a node failure would merely cause a brief hiccup. This would feel more like a period of increased latency than a full-blown outage. So there really needn't be much sacrifice of availability. However, there's also a sacrifice of partition tolerance. If the master is unable to communicate with any replica, the system can't serve requests. Also, the master is implemented as a collection of Paxos nodes; if these nodes are partitioned from one another, the entire system would grind to a halt. Since this is intended for intra-datacenter use, one could argue that a full network partition might be unlikely. (Depending on what sort of data center you hang out in.) But in CAP terms, it's possible, of course. (I base all this on the value-dependent chaining paper cited below.)

rescrv 15y ago
What you said was right on. I just wanted to add a few things. The coordinator is only involved for recovering from failures, so the cluster can still serve requests until server (non-coordinator) nodes start failing too. I would also add that if there is a intra-datacenter partition so severe as to violate HyperDex's failure assumptions, it will likely impact applications built on top of HyperDex as well. It would be necessary to survive such failures with an inter-datacenter system (which could be built on top of HyperDex).
- ypcx 15y ago
  Well first Thank you for providing another data storage possibility, and a great one at that. I'd just like to ask - the benchmarks where HyperDex beats even Redis - these are strictly clustered benchmarks - is that true? Or is the way HyperDex stores data so efficient, that it beats Redis even on a single core / single thread? Thanks!
  - gizzlon 15y ago
    Without more context graphs like that is pretty useless.. For all we know they just invented those numbers. I'm not saying they did, but you get my point.. Those numbers seem way to low for running on the same machine, and if not shouldn't the network be the bottleneck and show similar results for both? I'm sure there's a reasonable explanation, just as I'm sure they picket benchmarks that makes themselves look good.
nicktelford 15y ago
This sounds like a CP system to me. There's nothing wrong with that btw, I don't know why people are so reluctant to admit this. AP systems have some useful properties, but they're also (typically) more difficult to reason about. The "hiccups" you describe are periods of unavailability. The increased latency is caused by an element of the system waiting for the data to become available again, a totally valid strategy for coping with transient failures/partitions. Your argument about intra-datacenter partitions being unlikely are true, but they do happen. You also make a good point about such partitions also affecting client applications. Both of these are indicative of CP systems and, like I said: there's nothing wrong with that. Personally, I think both AP and CP distributed systems are equally interesting. What I consider a red flag is attempting to rationalize how a system "beats CAP".
- ericmoritz 15y ago
  That's what I was getting at. It's appears to be a CP system. I wish all new distributed databases would have a nice little badge that says, "CP", "AP" or "CA". Whenever they claim to be distributed but not subject to CAP, I automatically become skeptical. There's nothing wrong with a CP database; HBase is a CP database and it's quite popular.
- snewman 15y ago
  To be precise, I would argue this is actually a CA system. "C" because there are consistency guarantees, which are upheld even in the face of failures / partition. "A" because the system will continue making progress even after a node failure. What I called a "hiccup" can be made arbitrarily short, in principle at least. The system can work around failed nodes, it does not need to wait for them to be repaired. Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress.
  - jcdavis 15y ago
    > Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress. Not be able to progress means it is not available.
  - 15y ago
cstejerean 15y ago
"if these nodes are partitioned from one another, the entire system would grind to a halt." Keeping consistency at the expense of availability in the event of a partition is precisely what makes this a CP system.

by snewman 15y ago

rescrv 15y ago

What you said was right on. I just wanted to add a few things. The coordinator is only involved for recovering from failures, so the cluster can still serve requests until server (non-coordinator) nodes start failing too. I would also add that if there is a intra-datacenter partition so severe as to violate HyperDex's failure assumptions, it will likely impact applications built on top of HyperDex as well. It would be necessary to survive such failures with an inter-datacenter system (which could be built on top of HyperDex).

ypcx 15y ago
Well first Thank you for providing another data storage possibility, and a great one at that. I'd just like to ask - the benchmarks where HyperDex beats even Redis - these are strictly clustered benchmarks - is that true? Or is the way HyperDex stores data so efficient, that it beats Redis even on a single core / single thread? Thanks!
- gizzlon 15y ago
  Without more context graphs like that is pretty useless.. For all we know they just invented those numbers. I'm not saying they did, but you get my point.. Those numbers seem way to low for running on the same machine, and if not shouldn't the network be the bottleneck and show similar results for both? I'm sure there's a reasonable explanation, just as I'm sure they picket benchmarks that makes themselves look good.

nicktelford 15y ago

This sounds like a CP system to me. There's nothing wrong with that btw, I don't know why people are so reluctant to admit this. AP systems have some useful properties, but they're also (typically) more difficult to reason about. The "hiccups" you describe are periods of unavailability. The increased latency is caused by an element of the system waiting for the data to become available again, a totally valid strategy for coping with transient failures/partitions. Your argument about intra-datacenter partitions being unlikely are true, but they do happen. You also make a good point about such partitions also affecting client applications. Both of these are indicative of CP systems and, like I said: there's nothing wrong with that. Personally, I think both AP and CP distributed systems are equally interesting. What I consider a red flag is attempting to rationalize how a system "beats CAP".

ericmoritz 15y ago
That's what I was getting at. It's appears to be a CP system. I wish all new distributed databases would have a nice little badge that says, "CP", "AP" or "CA". Whenever they claim to be distributed but not subject to CAP, I automatically become skeptical. There's nothing wrong with a CP database; HBase is a CP database and it's quite popular.
snewman 15y ago
To be precise, I would argue this is actually a CA system. "C" because there are consistency guarantees, which are upheld even in the face of failures / partition. "A" because the system will continue making progress even after a node failure. What I called a "hiccup" can be made arbitrarily short, in principle at least. The system can work around failed nodes, it does not need to wait for them to be repaired. Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress.
- jcdavis 15y ago
  > Not "P", because if the network falls apart sufficiently, the system will generally not be able to progress. Not be able to progress means it is not available.
- 15y ago

cstejerean 15y ago

"if these nodes are partitioned from one another, the entire system would grind to a halt." Keeping consistency at the expense of availability in the event of a partition is precisely what makes this a CP system.