3 ms·
Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides http://hyperdex.org/slides/2012-04-13-HyperDex.pdf http://hyperdex.
by rdtsc 14y ago
Was trying to figure out where it lies in the CAP trade-off space. Found this set of slides
http://hyperdex.org/slides/2012-04-13-HyperDex.pdf http://hyperdex.org/slides/2012-04-13-HyperDex.pdf :
---
* Consistent: linearizable. GET returns latest PUT, always
* Available: in the presence of < f failures
* Partition-Tolerant: for partitions with < f nodes
---
So it seems they have strong C and a tunable trade-off between A and P ?
Then they take a jab at the popular interpretation of the CAP theorem and claim they have a work-around:
----
Working around the [CAP] theorem:
* Constrain the failure size
* Redirect clients to majority partition
* Profit: Retain all of C, A, P
Realistic for a modern data center
CAP misses the point The real tradeoff is between C, A, and Performance.
---
Not sure what they meant here. Anyone understood this better?
- rescrv 14y agoCAP, as stated, is a tautology. If you want to understand this better, please read our description of the CAP theorem (http://hyperdex.org/extras/ http://hyperdex.org/extras/). As formulated by Gilbert and Lynch, the CAP theorem really says, "If you must always read the latest value, and you must have any server serve any request, and clients are collocated with servers, then you cannot survive a partition." This is in the same class as, "If you separate all clients from servers, the clients cannot contact the servers." If, instead, you make assumptions about what kind of failures are actually possible, then you can do more. HyperDex assumes that if you have less than f nodes fail at any give point in time. A failure could be the process crashing, the machine being turned off, or a partition occurring within the datacenter. So long as this assumption is true, then HyperDex is able to survive the partition, make progress (remain available), and provide strong consistency while doing so. By making additional assumptions, we are able to make additional guarantees. This is not unlike what other engineers do (warning: massive simplification). A engineer building a bridge can assume the bridge can hold only so many cars and that each car has an upper bound on its weight. The product of these gives the entire weight the bridge must withstand. If the engineer builds the bridge to withstand the expected weight, plus some extra, then the bridge will work in all anticipated scenarios. It will not, of course, withstand bumper-to-bumper traffic consisting of 18-wheelers filled with lead bricks. But again, it doesn't have to.
- rdtsc 14y agoThanks for explaining. Your project looks very interesting. I will be definitely following it. Can you explain some more about how node failure are handled? And whether there is a way to tune that. I.e more about the phrase "less than f nodes fail at any give point in time". Is that tunable and how dynamic is that. (Set once or dynamic setting? Also set per running cluster, per name space, per key-value pair?) Also, how is this f number related to the total number of machines in the cluster and how is it related to the number of data replicas?
- rescrv 14y agof is the number of failures that the system can tolerate. Typical systems can tolerate f failures with either (f+1) or (2f + 1) replicas. HyperDex can tolerate f failures with f+1 replicas. This failure threshold is per region of the hyperspace. Within each region, servers are arranged in a chain. As servers fail they are removed from the chain. As new servers come into the cluster they are appended to the end of one or more chains. The "less than f nodes fail at any given point in time" translates to "at least one node in the chain is alive". Notice that chains are of length f+1 while only f nodes can fail, guaranteeing that one node is alive. This will be tunable parameter. Right now, it is set when a new space is created, but in an upcoming release this will be totally dynamic and changeable on a per-region (of the hyperspace) basis. I'd recommend f values of at most three. If failures are totally independent, this will be sufficient to last for millions of years. Thus, it is independent of the number of machines (once you have a sufficient number of machines).
- DennisP 14y ago"If failures are totally independent"...that's sometimes a big if, especially on cloud providers. I didn't see anything in the documentation about backups...what are the options?
- rescrv 14y agoMost of the decent providers will help you improve failure independence. I know first hand that Linode is very accommodating for this. I mention independent failures because f=3 is sufficient for this scenario for most. If there is correlation between failures and you cannot move servers around then you can compensate with a higher f. In a future release (within the next six months), we'll be adding support for consistent snapshots that guarantee that HyperDex can withstand more than f failures with a bounded amount of data loss. It's always possible to retrieve all data with an empty search and manually dump it into another form.