4 ms·

What's the clustering story these days? Last I checked, Redis Cluster still had lots of issues losing writes, and the design doesn't seem to have been revised s

What's the clustering story these days? Last I checked, Redis Cluster still had lots of issues losing writes, and the design doesn't seem to have been revised since Aphyr ran Jepsen tests on it back in 2013, making it (at least from my perspective) practically useless for anything that requires distributed consistency, which means most things that I look to use a distributed data store for. I've also been warned by fellow devs that much of the clustering logic is actually done in the client, and there's historically been a lack of mature clients for all languages. Even if you find a mature client, the complexity of the implementation implies that not all clients may behave identically, or may have different bugs. Then there's the issue of lock-in, where you become dependent on a specific client library and its development lifecycle. I don't know if all of this is true, but I also don't hear a lot of people talking about Redis Cluster these days. I know you can use Redis in master/slave mode via Sentinel + Twemproxy, though even this solution seems to have some issues with data consistency. Running all three also appears a lot more complex than an integrated system. I see a lot of comments implying that Redis is mainly used in single-node setups, so that might be where it shines?

techie128 8y ago
> I know you can use Redis in master/slave FYI - the current accepted PC term in the Valley is master/replica or leader/follower. master/slave was deemed too offensive not too long ago.
- matterhorn2000 8y ago
  Oh please.
- banachtarski 8y ago
  Deemed by who? I'm in the valley and this is the first time I'm encountering it. I need to leave.
- lessclue 8y ago
  Huh, what? Never heard of this DB replication "PC" nonsense before.
- xnxn 8y ago
  This is extremely obvious concern trolling. Please don't do that here.
  - techie128 8y ago
    Nope. It's not trolling if you state a fact.
    - xnxn 8y ago
      It absolutely is. Concern trolling means disingenuously expressing concern about an issue in order to undermine or derail genuine discussion, which you've done here, and your other remarks about "SJWs" lay bare your motives. I'm asking you to knock it off and discuss in good faith next time.
- asfasifn 8y ago
  I will say that while I know no one (except provocateurs though I've not encountered any) means ill by this terminology, it was jarring to listen to a recent presentation where an old white dude representing a company kept saying master and slave over and over. Sure, no one means ill, but if it's just a technical term, there's no need to keep it either and it would eliminate this grating irritation. This lingual shibboleth does at least signal a wider cultural acceptance of hierarchy and domination, but we already knew that living as we do in societies with stratified classes which also celebrate domination in sports, in history, the academy, and in corporate boardrooms. If some kind of alternative to that world sounds crazy to you, it's because that alternative was never presented as realistic. It is possible. EDIT: Also, congrats to Redis on 10 years!
  - lwkl 8y ago
    In Switzerland we had quasi slaves till the 1980s (Verdingkinder). This is horrible and the skin color of the victims was most of the time white. And many of them are now old white dudes. There are probably more old white dudes in Europe that suffered directly from slavery than there are Americans with Grandparents who were slaves. So please don‘t judge people by their skin color.
- enzanki_ars 8y ago
  I would strongly recommend that you read the author of Redis himself discussing this issue: http://antirez.com/news/122 http://antirez.com/news/122
  - icelancer 8y ago
    Antirez was pressured into adding aliases to soothe the crowd.
  - techie128 8y ago
    Doesn't change the fact that it is the currently accepted term in SV. See how viciously I was voted down and flagged for stating a fact and informing the OP about it. I don't care for the terminology myself and sure Antirez was pressured into it which is sad in itself. But atleast call a spade a spade. SJWs want to govern the terminology as well.
  - cmroanirgo 8y ago
    Thanks for this link. Some developers see more than others and can articulate differences very well. I wish I'd been scanning his blog for years! > "There is more: I believe that political correctness has a puritan root. As such it focuses on formalities, but actually it has a real root of prejudice against others. For instance Mark bullied me because I was not complying with his ideas, showing problems at accepting differences in the way people think. I believe that this environment is making it impossible to have important conversations."
antirez 8y ago
Hello! Redis Cluster has yet the same tradeoffs, because they were basically designed, and were not shortcomings. Btw Aphyr never tested Redis Cluster, and does not made any sense to test it because he tests things for strong guarantees so it should test Redis-CP instead, for instance, a consistent store for Redis implemented into a module using Raft. I believe that for the Redis use case the cluster tradeoffs make sense because: * Best effort consistency in the practice work quite well, even if it does not have any guarantee, but does certain things to avoid losing writes in trivial ways. * If you want to cluster Redis, you want Redis, not a cluster that automatically becomes a lot more slower, memory hungry, and so forth. So replication should be asynchronous. However what I may change in the future, and there are plans for that, is to add a failover strategy that does not just pick the replica that is more ahead in terms of received writes: but even stop the failover if there isn't a majority of slaves reachable. This improves certain properties, and if well orchestrated can also show strong properties if writes are acknowledged only after transferred to the majority of slaves. Redis Cluster is used in many organizations right now. The next step is to improve it (in different ways than having strong properties mostly), and provide an official proxy for it.
- Jupe 8y ago
  Interesting approach! Reminds of an option my group considered a couple years ago, where we purposely didn't set up read replicas, but considered failure to connect/read of the master as a 'C' failure. Hashing keys to multiple masters and we had "highly available" redis with stronger consistency. We ended up not using it, but I always wanted to revisit the pattern. BTW, thanks for such a wonderful piece of software!
  - sparkcc 8y ago
    Yes. Thanks for this software ! BTW, Without a critical mass of users regularly vouching, the data isn't much help. if someone is happily using the package in production, that won't tick any download numbers even though it's arguably the thing anyone considering a package would really want to know.
- atombender 8y ago
AtlasBarfed 8y ago
SighThe responses to your question (systems falling over on Jepsen) are already the most depressing kind: - that's not our use case (then why are you advertising distribution???) - they didn't use this feature (WELL THEN, SHOW THAT FEATURE PASSING THE TESTS) - we're not that kind of distributed (oh, you mean the one you can use?) Jepsen has been the first step for trusting anything that involves data and distribution. Everything is going to fail under his hard stare, it just comes down to how much BS smokescreening is done by the purveyors of the software to judge how to trust it.

techie128 8y ago

> I know you can use Redis in master/slave FYI - the current accepted PC term in the Valley is master/replica or leader/follower. master/slave was deemed too offensive not too long ago.

matterhorn2000 8y ago
Oh please.
banachtarski 8y ago
Deemed by who? I'm in the valley and this is the first time I'm encountering it. I need to leave.
lessclue 8y ago
Huh, what? Never heard of this DB replication "PC" nonsense before.
xnxn 8y ago
This is extremely obvious concern trolling. Please don't do that here.
- techie128 8y ago
  Nope. It's not trolling if you state a fact.
  - xnxn 8y ago
    It absolutely is. Concern trolling means disingenuously expressing concern about an issue in order to undermine or derail genuine discussion, which you've done here, and your other remarks about "SJWs" lay bare your motives. I'm asking you to knock it off and discuss in good faith next time.
asfasifn 8y ago
I will say that while I know no one (except provocateurs though I've not encountered any) means ill by this terminology, it was jarring to listen to a recent presentation where an old white dude representing a company kept saying master and slave over and over. Sure, no one means ill, but if it's just a technical term, there's no need to keep it either and it would eliminate this grating irritation. This lingual shibboleth does at least signal a wider cultural acceptance of hierarchy and domination, but we already knew that living as we do in societies with stratified classes which also celebrate domination in sports, in history, the academy, and in corporate boardrooms. If some kind of alternative to that world sounds crazy to you, it's because that alternative was never presented as realistic. It is possible. EDIT: Also, congrats to Redis on 10 years!
- lwkl 8y ago
  In Switzerland we had quasi slaves till the 1980s (Verdingkinder). This is horrible and the skin color of the victims was most of the time white. And many of them are now old white dudes. There are probably more old white dudes in Europe that suffered directly from slavery than there are Americans with Grandparents who were slaves. So please don‘t judge people by their skin color.
enzanki_ars 8y ago
I would strongly recommend that you read the author of Redis himself discussing this issue: http://antirez.com/news/122 http://antirez.com/news/122
- icelancer 8y ago
  Antirez was pressured into adding aliases to soothe the crowd.
- techie128 8y ago
  Doesn't change the fact that it is the currently accepted term in SV. See how viciously I was voted down and flagged for stating a fact and informing the OP about it. I don't care for the terminology myself and sure Antirez was pressured into it which is sad in itself. But atleast call a spade a spade. SJWs want to govern the terminology as well.
- cmroanirgo 8y ago
  Thanks for this link. Some developers see more than others and can articulate differences very well. I wish I'd been scanning his blog for years! > "There is more: I believe that political correctness has a puritan root. As such it focuses on formalities, but actually it has a real root of prejudice against others. For instance Mark bullied me because I was not complying with his ideas, showing problems at accepting differences in the way people think. I believe that this environment is making it impossible to have important conversations."

antirez 8y ago

Hello! Redis Cluster has yet the same tradeoffs, because they were basically designed, and were not shortcomings. Btw Aphyr never tested Redis Cluster, and does not made any sense to test it because he tests things for strong guarantees so it should test Redis-CP instead, for instance, a consistent store for Redis implemented into a module using Raft. I believe that for the Redis use case the cluster tradeoffs make sense because: * Best effort consistency in the practice work quite well, even if it does not have any guarantee, but does certain things to avoid losing writes in trivial ways. * If you want to cluster Redis, you want Redis, not a cluster that automatically becomes a lot more slower, memory hungry, and so forth. So replication should be asynchronous. However what I may change in the future, and there are plans for that, is to add a failover strategy that does not just pick the replica that is more ahead in terms of received writes: but even stop the failover if there isn't a majority of slaves reachable. This improves certain properties, and if well orchestrated can also show strong properties if writes are acknowledged only after transferred to the majority of slaves. Redis Cluster is used in many organizations right now. The next step is to improve it (in different ways than having strong properties mostly), and provide an official proxy for it.

Jupe 8y ago
Interesting approach! Reminds of an option my group considered a couple years ago, where we purposely didn't set up read replicas, but considered failure to connect/read of the master as a 'C' failure. Hashing keys to multiple masters and we had "highly available" redis with stronger consistency. We ended up not using it, but I always wanted to revisit the pattern. BTW, thanks for such a wonderful piece of software!
- sparkcc 8y ago
  Yes. Thanks for this software ! BTW, Without a critical mass of users regularly vouching, the data isn't much help. if someone is happily using the package in production, that won't tick any download numbers even though it's arguably the thing anyone considering a package would really want to know.
atombender 8y ago

AtlasBarfed 8y ago

SighThe responses to your question (systems falling over on Jepsen) are already the most depressing kind: - that's not our use case (then why are you advertising distribution???) - they didn't use this feature (WELL THEN, SHOW THAT FEATURE PASSING THE TESTS) - we're not that kind of distributed (oh, you mean the one you can use?) Jepsen has been the first step for trusting anything that involves data and distribution. Everything is going to fail under his hard stare, it just comes down to how much BS smokescreening is done by the purveyors of the software to judge how to trust it.