6 ms·

Redis cluster, no longer vaporware

522 points by untothebreach 12y ago

ryanpetrich 12y ago
Can't wait to see it get the Jepsen treatment.
- jrallison 12y ago
  It won't do well, but at least Antirez is up front about the shortcomings. The article itself states that Redis Cluster will lose acknowledged writes during partitions (no P), will be eventually consistent (no C), and won't be available during partitions (no A).
  - antirez 12y ago
    Yep, Jepsen is more suitable to check systems that claim either linerizability, or at least write safety, during partitions. I guess that a modified version of Jepsen could be used in order to validate the failure modes or to discover other unexpected ones that at human inspection look easy to reproduce in actual production environments. Also I don't know if Jepsen is good at this, but in theory it could be instrumented in order to check how good the implementation is, which is, even if it is not designed for write safety during partitions, how better the countermeasures are working?
    - bkirwi 12y ago
      In theory, Jepsen or a Jepsen-like system should be able to check any of these failure modes. On the other hand, it sounds like Redis Cluster offers few hard guarantees; instead, it promises that failures should be rare 'in practice'. Which is a fine thing for a tool to do, of course, but it makes things less amenable to the kind of stress-testing Jepsen does -- since running inside Jepsen's little universe is about as far from normal operation as you can get. If you already know that a system can fail in a certain way, getting Jepsen to reproduce that failure tells you very little. If you'd like to make this kind of testing possible, it would be useful to state as many 'positive' rules as possible, which Redis Cluster should always respect -- things like "if a majority of nodes are fully connected, they should always accept writes" and "an unpartitioned cluster should always agree on the same value" -- alongside the documentation on ways it might fail. This way, clients can be assured of the 'bare minimum' that the system supports, and tools like Jepsen can give you more useful information.
      - antirez 12y ago
        Oh, there are definitely hard rules like that. For example a majority partition never accepts queries, and when there are no partitions at all Redis Cluster guarantees to converge on a single value for each key, and to a single view of the cluster configuration. I'll try to document better this things, but basically they arise from the simple algorithm that makes the configuration eventually consistent.
        bkirwi 12y ago
        Neat; that should be very useful. It's great to see that Redis has a official story for clustering / failover out; like you said in the post, the worst distributed systems are the ones you have to rewrite every single time. It's going to be interesting watching this evolve.
  - swartkrans 12y ago
    Seems like given the above problems you would only want to use this with non-critical data of which you didn't care a great deal about its integrity. This doesn't describe a lot of use cases. It's not too hard to get memcached running for cluster of machines, nor is it really all that difficult to get started with cassandra for basic applications, although I haven't done it. I've heard ReThinkDB wants to be a serious contender and that they've tried to solve these problems, but I haven't read about how great it is nor have I heard anyone use it in production so I don't know where it stands.
    - antirez 12y ago
      Note that systems with asynchronous replication and normal failover procedures are all subject to the same rules as Redis Cluster, and I believe there is a great deal of people running RDBMs in this setup. So IMHO actually there are a lot of use cases.
      - IgorPartola 12y ago
        From what I understand, there are two types of systems really (in terms of CAP): those that do async replication and those that do sync replication. Async replicated systems can be (but are not necessarily) AP: each system can process requests independently of the other, and they'll get consistent eventually (how they arrive at this consistency is different between systems, and is not always "correct" for the general use case). Sync replicated systems can be (but are not necessarily) CA: as long as replication is not broken they are available and consistent. These systems have the upside in that they allow you to serve read-only data during partitions, but have slow writes because of the sync replication. Both types can have features removed, for whatever reason. For example, you could have a sync replicated system that doesn't serve any requests when the replication is broken. Or you could have an async system that doesn't provide a mechanism to correct inconsistencies (a la MySQL async replication). What you describe sounds like an imperfect version of the async system: not highly available, not consistent, and not partition tolerant. Given that better AP systems exist, ones that actually provide things like availability during partitions and eventual consistency mechanisms/guarantees, will Redis Cluster eventually support these features? Edit: please don't take any of this as criticism. Redis is an awesome product that I use and rely on often. Thanks for all the great work.
- jbinto 12y ago
  I didn't get the context, so I looked it up: http://aphyr.com/tags/Jepsen http://aphyr.com/tags/Jepsen > This article is part of Jepsen, a series on network partitions. Covers MongoDB, redis, RabbitMQ, among others. All articles start with "call me maybe", thus the Jepsen reference.
  - deleted 12y ago
    [deleted]
- rurounijones 12y ago
  Would love to see Jepsen thrown at Apache Storm.
chrisan 12y ago
Anyone know how AWS clusters Redis? I had alwas just assumed Redis released this a while ago and that is why AWS started to support Redis along with Memcache in their clustered ElastiCache product http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-elasticache-for-redis/ http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-...
- ihsw 12y ago
  That's failover -- essentially Redis Sentinel. Redis Cluster on the other hand orchestrates multiple redis instances to function as one, thereby preventing the need for redis servers that are very large and preventing the need for presharding[1]. [1] http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/ http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigsl...
  - toomuchtodo 12y ago
    Redis Cluster appears to be similar to using memcached with Facebook's mcrouter [1]. [1] https://github.com/facebook/mcrouter https://github.com/facebook/mcrouter
- mfenniak 12y ago
  I have very little evidence to support this, but they probably use twemproxy. twemproxy is a proxy for redis that automatically manages routing commands to multiple redis servers efficiently. They definitely don't use redis cluster, because redis cluster requires explicit client-side support. That support is only available in a limited number of clients today.
  - alphadevx 12y ago
    Never used it, but as far as I know twemproxy does not support the select command to select another Redis database on the current connection, which would be a good indicator of it being in use at Amazon or not.
  - seiji 12y ago
    Nope, they don't (and neither should anybody else... twemproxy is weird).
alphadevx 12y ago
Single-master Redis and replication overhead of current Redis architecture not ideal for getting Redis beyond the "it's a great cache" use case, so I'm been looking forward to this.
itamarhaber 12y ago
Congratulations @antirez and the Redis community - this is a key step forward for this amazingly useful database!
peterwwillis 12y ago
@antirez, what would you say is the most useful lesson you've learned throughout this process?
- antirez 12y ago
  Argument with people just with technical arguments, don't get into Twitter fights, and at the end of the day, use the feedbacks as a learning process, but do what you believe makes sense. So a social lesson, more than technological.
  - legedemon 12y ago
    Thanks for this lesson! As much as I love using Redis, this learning will help me way more than Redis has done.
merlin83 12y ago
I created a Fig environment in case anyone is interested in giving it a spin. https://github.com/merlin83/rediscluster-docker https://github.com/merlin83/rediscluster-docker
jpgvm 12y ago
I am going to be honest, I am somewhat disappointed. I was hoping for a CP system with sharding + replication. Without which we can't have things like multi-key atomic operations etc... Will wait and see though, maybe things are better than they seem.
- antirez 12y ago
  I'm pretty sure that if you are a Redis user, whatever you hate or like the current Redis Cluster design, what you never wanted, is a CP system. CP systems need agreement at every query so the latency and OPS/sec figures are not suitable for Redis-alike use cases. Actually with an existing CP system you can easily build the same Redis data structures since CP systems are linearizable so you can have a CP shell that internally runs a Redis kernel. Imagine the CP system is Raft, you just consider Redis your internal state machine, and every operation that is committed is applied to Redis just sending it the command.
  - jpgvm 12y ago
    That is actually exactly what I wanted. :) I have done something similar with Zookeeper recently (through I wrote it in Ruby so it's terribly slow). You would be surprised how little CP stores are actually available. There is an abundancy of AP stores but very very few useful CP stores, especially ones with shared-nothing architecture. You don't need for every query to traverse the consensus algorithm. It's sufficient to use the consensus algorithm to agree on the master shards for each chunk of data and have sync replication. Considering Redis is designed for 100% in-memory workloads if you are operating on a low latency network this is more than fine. Especially because this behavior would only be required for writes, reads could be served from either the master or a sync replica (and in theory if you had configurable read consistency a possibly out of date async replica). I guess what I am sad about is just the lack of CP stores that are useful right now, I probably just need to suck it up and start writing my own.
thorway-redis 12y ago
Your communication with your users is wonderful. I'm grateful for it, and appreciate how much effort goes into it. Thank you.
brianpgordon 12y ago
> From the point of view of distributed databases, Redis Cluster provides a limited amount of availability during partitions, and a weak form of consistency. Basically it is neither a CP nor an AP system. In other words, Redis Cluster does not achieve the theoretical limits of what is possible with distributed systems, in order to gain certain real world properties. @antirez Could you elaborate on what these properties are and your thinking behind why they're more important than AP?
joshbaptiste 12y ago
Wonder is Antirez is considering starting a commercial Redis alternative providing commercial support a la Nginx.
- antirez 12y ago
  The world if full of possibilities... for now to be honest I focused a lot in the technical / community side, and the sponsorship model helped a lot with that. However for 10 years before starting this project I was self-employed and started two companies. So yes, doing companies is generally tempting for me, just in the case of Redis so far it was more an evolution of an hacking session, lasting 6 years.
abluecloud 12y ago
>stopped and stopped

522 points by untothebreach 12y ago

ryanpetrich 12y ago

Can't wait to see it get the Jepsen treatment.

jrallison 12y ago
It won't do well, but at least Antirez is up front about the shortcomings. The article itself states that Redis Cluster will lose acknowledged writes during partitions (no P), will be eventually consistent (no C), and won't be available during partitions (no A).
- antirez 12y ago
  Yep, Jepsen is more suitable to check systems that claim either linerizability, or at least write safety, during partitions. I guess that a modified version of Jepsen could be used in order to validate the failure modes or to discover other unexpected ones that at human inspection look easy to reproduce in actual production environments. Also I don't know if Jepsen is good at this, but in theory it could be instrumented in order to check how good the implementation is, which is, even if it is not designed for write safety during partitions, how better the countermeasures are working?
  - bkirwi 12y ago
    In theory, Jepsen or a Jepsen-like system should be able to check any of these failure modes. On the other hand, it sounds like Redis Cluster offers few hard guarantees; instead, it promises that failures should be rare 'in practice'. Which is a fine thing for a tool to do, of course, but it makes things less amenable to the kind of stress-testing Jepsen does -- since running inside Jepsen's little universe is about as far from normal operation as you can get. If you already know that a system can fail in a certain way, getting Jepsen to reproduce that failure tells you very little. If you'd like to make this kind of testing possible, it would be useful to state as many 'positive' rules as possible, which Redis Cluster should always respect -- things like "if a majority of nodes are fully connected, they should always accept writes" and "an unpartitioned cluster should always agree on the same value" -- alongside the documentation on ways it might fail. This way, clients can be assured of the 'bare minimum' that the system supports, and tools like Jepsen can give you more useful information.
    - antirez 12y ago
      Oh, there are definitely hard rules like that. For example a majority partition never accepts queries, and when there are no partitions at all Redis Cluster guarantees to converge on a single value for each key, and to a single view of the cluster configuration. I'll try to document better this things, but basically they arise from the simple algorithm that makes the configuration eventually consistent.
      - bkirwi 12y ago
        Neat; that should be very useful. It's great to see that Redis has a official story for clustering / failover out; like you said in the post, the worst distributed systems are the ones you have to rewrite every single time. It's going to be interesting watching this evolve.
- swartkrans 12y ago
  Seems like given the above problems you would only want to use this with non-critical data of which you didn't care a great deal about its integrity. This doesn't describe a lot of use cases. It's not too hard to get memcached running for cluster of machines, nor is it really all that difficult to get started with cassandra for basic applications, although I haven't done it. I've heard ReThinkDB wants to be a serious contender and that they've tried to solve these problems, but I haven't read about how great it is nor have I heard anyone use it in production so I don't know where it stands.
  - antirez 12y ago
    Note that systems with asynchronous replication and normal failover procedures are all subject to the same rules as Redis Cluster, and I believe there is a great deal of people running RDBMs in this setup. So IMHO actually there are a lot of use cases.
    - IgorPartola 12y ago
      From what I understand, there are two types of systems really (in terms of CAP): those that do async replication and those that do sync replication. Async replicated systems can be (but are not necessarily) AP: each system can process requests independently of the other, and they'll get consistent eventually (how they arrive at this consistency is different between systems, and is not always "correct" for the general use case). Sync replicated systems can be (but are not necessarily) CA: as long as replication is not broken they are available and consistent. These systems have the upside in that they allow you to serve read-only data during partitions, but have slow writes because of the sync replication. Both types can have features removed, for whatever reason. For example, you could have a sync replicated system that doesn't serve any requests when the replication is broken. Or you could have an async system that doesn't provide a mechanism to correct inconsistencies (a la MySQL async replication). What you describe sounds like an imperfect version of the async system: not highly available, not consistent, and not partition tolerant. Given that better AP systems exist, ones that actually provide things like availability during partitions and eventual consistency mechanisms/guarantees, will Redis Cluster eventually support these features? Edit: please don't take any of this as criticism. Redis is an awesome product that I use and rely on often. Thanks for all the great work.
jbinto 12y ago
I didn't get the context, so I looked it up: http://aphyr.com/tags/Jepsen http://aphyr.com/tags/Jepsen > This article is part of Jepsen, a series on network partitions. Covers MongoDB, redis, RabbitMQ, among others. All articles start with "call me maybe", thus the Jepsen reference.
- deleted 12y ago
  [deleted]
rurounijones 12y ago
Would love to see Jepsen thrown at Apache Storm.

chrisan 12y ago

Anyone know how AWS clusters Redis? I had alwas just assumed Redis released this a while ago and that is why AWS started to support Redis along with Memcache in their clustered ElastiCache product http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-elasticache-for-redis/ http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-...

ihsw 12y ago
That's failover -- essentially Redis Sentinel. Redis Cluster on the other hand orchestrates multiple redis instances to function as one, thereby preventing the need for redis servers that are very large and preventing the need for presharding[1]. [1] http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/ http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigsl...
- toomuchtodo 12y ago
  Redis Cluster appears to be similar to using memcached with Facebook's mcrouter [1]. [1] https://github.com/facebook/mcrouter https://github.com/facebook/mcrouter
mfenniak 12y ago
I have very little evidence to support this, but they probably use twemproxy. twemproxy is a proxy for redis that automatically manages routing commands to multiple redis servers efficiently. They definitely don't use redis cluster, because redis cluster requires explicit client-side support. That support is only available in a limited number of clients today.
- alphadevx 12y ago
  Never used it, but as far as I know twemproxy does not support the select command to select another Redis database on the current connection, which would be a good indicator of it being in use at Amazon or not.
- seiji 12y ago
  Nope, they don't (and neither should anybody else... twemproxy is weird).

alphadevx 12y ago

Single-master Redis and replication overhead of current Redis architecture not ideal for getting Redis beyond the "it's a great cache" use case, so I'm been looking forward to this.

itamarhaber 12y ago

Congratulations @antirez and the Redis community - this is a key step forward for this amazingly useful database!

peterwwillis 12y ago

@antirez, what would you say is the most useful lesson you've learned throughout this process?

antirez 12y ago
Argument with people just with technical arguments, don't get into Twitter fights, and at the end of the day, use the feedbacks as a learning process, but do what you believe makes sense. So a social lesson, more than technological.
- legedemon 12y ago
  Thanks for this lesson! As much as I love using Redis, this learning will help me way more than Redis has done.

merlin83 12y ago

I created a Fig environment in case anyone is interested in giving it a spin. https://github.com/merlin83/rediscluster-docker https://github.com/merlin83/rediscluster-docker

jpgvm 12y ago

I am going to be honest, I am somewhat disappointed. I was hoping for a CP system with sharding + replication. Without which we can't have things like multi-key atomic operations etc... Will wait and see though, maybe things are better than they seem.

antirez 12y ago
I'm pretty sure that if you are a Redis user, whatever you hate or like the current Redis Cluster design, what you never wanted, is a CP system. CP systems need agreement at every query so the latency and OPS/sec figures are not suitable for Redis-alike use cases. Actually with an existing CP system you can easily build the same Redis data structures since CP systems are linearizable so you can have a CP shell that internally runs a Redis kernel. Imagine the CP system is Raft, you just consider Redis your internal state machine, and every operation that is committed is applied to Redis just sending it the command.
- jpgvm 12y ago
  That is actually exactly what I wanted. :) I have done something similar with Zookeeper recently (through I wrote it in Ruby so it's terribly slow). You would be surprised how little CP stores are actually available. There is an abundancy of AP stores but very very few useful CP stores, especially ones with shared-nothing architecture. You don't need for every query to traverse the consensus algorithm. It's sufficient to use the consensus algorithm to agree on the master shards for each chunk of data and have sync replication. Considering Redis is designed for 100% in-memory workloads if you are operating on a low latency network this is more than fine. Especially because this behavior would only be required for writes, reads could be served from either the master or a sync replica (and in theory if you had configurable read consistency a possibly out of date async replica). I guess what I am sad about is just the lack of CP stores that are useful right now, I probably just need to suck it up and start writing my own.

thorway-redis 12y ago

Your communication with your users is wonderful. I'm grateful for it, and appreciate how much effort goes into it. Thank you.

brianpgordon 12y ago

> From the point of view of distributed databases, Redis Cluster provides a limited amount of availability during partitions, and a weak form of consistency. Basically it is neither a CP nor an AP system. In other words, Redis Cluster does not achieve the theoretical limits of what is possible with distributed systems, in order to gain certain real world properties. @antirez Could you elaborate on what these properties are and your thinking behind why they're more important than AP?

joshbaptiste 12y ago

Wonder is Antirez is considering starting a commercial Redis alternative providing commercial support a la Nginx.

antirez 12y ago
The world if full of possibilities... for now to be honest I focused a lot in the technical / community side, and the sponsorship model helped a lot with that. However for 10 years before starting this project I was self-employed and started two companies. So yes, doing companies is generally tempting for me, just in the case of Redis so far it was more an evolution of an hacking session, lasting 6 years.

abluecloud 12y ago

>stopped and stopped