3 ms·

This analysis does a good job of laying out what the software does, but there's no reason why Redis needs to work in this unexpected and confusing way. Handlin

by bri3d 12y ago

This analysis does a good job of laying out what the software does, but there's no reason why Redis needs to work in this unexpected and confusing way. Handling replication failures (especially those due to crashed nodes) in in-memory databases is hardly a new problem and there are formally described protocols and solutions available which do not require permanent storage and avoid many of the drawbacks of the Redis Sentinel. IMO Redis throughout its life has had an identity issue: is it an in-memory cache, a 'data structures server' like the documentation used to say, a persistent key-set/key-value/key-? database, or a clustered/distributed systems solution? Like MongoDB, Redis has ended up used in a lot of inappropriate environments because of its confusing, mixed messaging. I personally think Redis is great as a "data structures server" but would never trust it as a single source of truth or persistent store. When I've used it in the past to perform and cache set operations (for example, generating and caching view-privacy lists using set intersections), it's been excellent, but we've frequently had to re-populate the Redis store from our source of truth due to bugs or hardware outage / network partition scenarios which Redis doesn't yet handle well.

resca79 12y ago
thanks for sharing your experience. Did you used persistence features of redis or just the configuration base? like memcache?
- bri3d 12y ago
  In the specific privacy-list usecase, were very early users of Redis (2009 timeframe). We used RDB persistence for a while but eventually our load got too high for the fork()-and-snapshot model to work for us. When AOF was introduced we tried using it but the initial few releases were extremely buggy to the point that it lost data and confused us more than it provided persistence. We eventually decided that since we were using Redis as a glorified cache anyway (not a source of truth), we might as well leave it to the in-memory only configuration and make our state-restoration system faster and more robust. I've never used Redis as a source of truth / persistent data store because when I'm evaluating a database to use as a persistent source of truth, it needs to have a strong story around storage durability, replication, node failure recovery, and CAP tradeoffs / partition tolerance specifically. NOTE: I've used Redis since 2009, of course (still do!), and was just expounding on the anecdote from my parent post. All that's changed in terms of persistence for my use cases is that AOF tends to be fairly workable and not very buggy anymore, and that using a snapshot-slave makes the RDB snapshot approach more workable. Because the replication/clustering story has never offered any good guarantees around node failure or partition tolerance, I haven't used Redis as a source of truth - I simply use the persistence features to prevent needing to repopulate/re-warm my cache and list+graph store in the event of an outage.
  - resca79 12y ago
    I have figured out what you say redis, but has changed a lot since 2009. Basically I'm a rails developer so my I usually prefer ActiveRecord.. Mysql or Postgres. However I have noticed that redis is becoming a very good alternative to the persistent storage in term of hot cache(like twitter) when you have heavy writes not only for cache the data after the first read(like rails in memcache). Another interesting feature, in the next release, is the support for geospatial
antirez 12y ago
Hello bri3d, it's true that Redis is at the intersection of many use cases: for some it is appropriate, for some it is not. But about the memory VS disk debate, we are making progresses in this area. The thing is: Redis can be used as a purely in-memory system, but not if you enable replication, since replication used to have an issue, it needed to persist on disk in order to synchronize the first time with the replica. At some point we made this issue a bit more manageable, by implementing partial resynchronizations. However still for the first SYNC to persist was needed, and even after reconnections if there was not enough "backlog". So when people said "I want to use Redis in a master-slave setup, but, with persistence turned off, and I still want my data back after restarts", I used to reply "turn on persistence instead, since anyway for replication you NEED to save on disk". However, now there is a new thing that is changing my mind. A few days ago I finished a beta of diskless replication, this will be merged into unstable in a matter of days. Suddenly Redis no longer needs disks to replicate with its slaves. Now to have a master-slave setup with zero disk involvement is a real thing, so I'm trying to understand how I provide features to make this use case simpler to support. One feature which is trivial to add and can solve many problems, is protected restarts. Basically, if configured to do so, a master can return back online after a restart in a protected mode, where commands are not accepted, but instead an error is reported. This mode can be escaped only using SLAVEOF to configure the instance as a master, or as a slave. This way, slaves can't replicate from a rebooted master, and their data set is not lost. Similarly, Redis Sentinel will see the master as failing (since it replies with errors to PIGNs), so will perform a failover and a slave will be elected. The old slave will be reconfigured as a slave, and this in turn will undo the protected mode. Given it's a few lines of code looks like a promising solution that users can have ASAP.

by bri3d 12y ago

resca79 12y ago

thanks for sharing your experience. Did you used persistence features of redis or just the configuration base? like memcache?

bri3d 12y ago
In the specific privacy-list usecase, were very early users of Redis (2009 timeframe). We used RDB persistence for a while but eventually our load got too high for the fork()-and-snapshot model to work for us. When AOF was introduced we tried using it but the initial few releases were extremely buggy to the point that it lost data and confused us more than it provided persistence. We eventually decided that since we were using Redis as a glorified cache anyway (not a source of truth), we might as well leave it to the in-memory only configuration and make our state-restoration system faster and more robust. I've never used Redis as a source of truth / persistent data store because when I'm evaluating a database to use as a persistent source of truth, it needs to have a strong story around storage durability, replication, node failure recovery, and CAP tradeoffs / partition tolerance specifically. NOTE: I've used Redis since 2009, of course (still do!), and was just expounding on the anecdote from my parent post. All that's changed in terms of persistence for my use cases is that AOF tends to be fairly workable and not very buggy anymore, and that using a snapshot-slave makes the RDB snapshot approach more workable. Because the replication/clustering story has never offered any good guarantees around node failure or partition tolerance, I haven't used Redis as a source of truth - I simply use the persistence features to prevent needing to repopulate/re-warm my cache and list+graph store in the event of an outage.
- resca79 12y ago
  I have figured out what you say redis, but has changed a lot since 2009. Basically I'm a rails developer so my I usually prefer ActiveRecord.. Mysql or Postgres. However I have noticed that redis is becoming a very good alternative to the persistent storage in term of hot cache(like twitter) when you have heavy writes not only for cache the data after the first read(like rails in memcache). Another interesting feature, in the next release, is the support for geospatial

antirez 12y ago

Hello bri3d, it's true that Redis is at the intersection of many use cases: for some it is appropriate, for some it is not. But about the memory VS disk debate, we are making progresses in this area. The thing is: Redis can be used as a purely in-memory system, but not if you enable replication, since replication used to have an issue, it needed to persist on disk in order to synchronize the first time with the replica. At some point we made this issue a bit more manageable, by implementing partial resynchronizations. However still for the first SYNC to persist was needed, and even after reconnections if there was not enough "backlog". So when people said "I want to use Redis in a master-slave setup, but, with persistence turned off, and I still want my data back after restarts", I used to reply "turn on persistence instead, since anyway for replication you NEED to save on disk". However, now there is a new thing that is changing my mind. A few days ago I finished a beta of diskless replication, this will be merged into unstable in a matter of days. Suddenly Redis no longer needs disks to replicate with its slaves. Now to have a master-slave setup with zero disk involvement is a real thing, so I'm trying to understand how I provide features to make this use case simpler to support. One feature which is trivial to add and can solve many problems, is protected restarts. Basically, if configured to do so, a master can return back online after a restart in a protected mode, where commands are not accepted, but instead an error is reported. This mode can be escaped only using SLAVEOF to configure the instance as a master, or as a slave. This way, slaves can't replicate from a rebooted master, and their data set is not lost. Similarly, Redis Sentinel will see the master as failing (since it replies with errors to PIGNs), so will perform a failover and a slave will be elected. The old slave will be reconfigured as a slave, and this in turn will undo the protected mode. Given it's a few lines of code looks like a promising solution that users can have ASAP.