4 ms·

Oh, I didn't expected to see a patch release posted here. As I wrote in the announce this patch release is a bit special as it adds a few new features so at lea

by antirez 13y ago

Oh, I didn't expected to see a patch release posted here. As I wrote in the announce this patch release is a bit special as it adds a few new features so at least there is something to explore. Btw ask me anything about this release if something is not clear, I'll try to reply at my best.

ksec 13y ago
Just want to say, Thank You for Redis.
- antirez 13y ago
  Thank you ksec, without a great community there is no fun.
- Jakehp 13y ago
  +1. I just started learning and using it a few weeks ago; it's been a lot of fun.
leobelle 13y ago
Thanks for answering questions, any comments or progress on aphyr's complaints about using proven algorithms for Redis replication? [1] [1] http://aphyr.com/posts/307-call-me-maybe-redis-redux http://aphyr.com/posts/307-call-me-maybe-redis-redux
- antirez 13y ago
  Given that Apyhr's investigation in the blog post you cite does not involve anything implemented or planned for Redis, I'm not able to reply to the question. Maybe you fell in the mediatic trap of considering that discussion about Redis turning into a CP system using the proposed algorithm. If you re-read the post, Aphyr itself explains that this is not Redis Cluster. If you are curious about what triggered the post, is my claim that WAIT per se as a primitive does not mean CP or any other kind of consistency, but that it is defined by the rest of the system. To prove my point, I orchestrated a toy and god-driven distributed system where WAIT turned into a strong consistent primitive. The point is that you can replace the "God" part with other stuff to get it real-world. Apyhr modeled it and found that if you don't read the same way you write (via agreement) the system is inconsistent when you consider reads, but I think this was common knowledge. Every CP system in the base case needs to follow the same path for writes and reads. However most of the times you can invent optimizations to speedup reads. p.s. Redis replication is asynchronous by default. The WAIT primitive in Redis unstable does just what it claims, that is, to return an acknowledgment when the specified number of replicas received some data. The practical effect is that you can make writes safer for a (big) latency cost.
  - lucian1900 13y ago
    You answered a different question. Are you using a verified consensus algorithm? If not, why are you making a distributed database? You're not inspiring confidence in using anything but single node Redis.
    - redbad 13y ago
      It seems that antirez believes he's allowing Redis to approach AP and/or CP semantics by providing primitives that users can compose in ad-hoc ways according to their requirements. Unfortunately, that's not a valid methodology to achieve availability or (especially) consistency.
      - antirez 13y ago
        In Redis Cluster there is nothing you can do to tune availability or consistency. The only thing you can tune is write "safety" using WAIT, but actually most users should never use it since the system is mostly designed to be low-latency. Redis Cluster is not AP nor CP. It resembles more AP since it has no strong consistency, however it is not able to reach "A" of AP that is very strict. Like AP systems have some form of consistency, but not "C" of CAP. Redis does this for "A" as well, it has some form of availability, but not as strict as "A" of CAP. Why this? You'll notice that there are not AP database systems with the powerful commands of Redis, and that there are not CP systems with the latency of Redis. So it is a very opinionated trade off to get both sacrificing other stuff (especially "A").
        leobelle 13y ago
        If it's not AP or CP, then what is the value? Cassandra is eventually consistent. Consistency is pretty important. It seems like Redis Cluster doesn't have consistency, nor availability, nor partition tolerance. Maybe I misunderstand it though.
        antirez 13y ago
        > Cassandra is eventually consistent. Consistency is pretty important. Your above statement unfortunately has no match in distributed systems theory or practice. Basically eventually consistent means that the system is not consistent from the point of view of CAP, so it does not feature strong consistency. As an alternative eventual consistency means: our contract with the user is, that, at least, when there are no partitions, the system will no show split-brain conditions, so a given information will converge to a given value for all the nodes. This alone is a contract with the user, but does not tell a lot, it just tells you: 1) No strong consistency, sorry. 2) No split-brain conditions when the partitions heal. So "It features eventual consistency, consistency is important" statement, I'm afraid, means that you don't have the tradeoffs about distributed systems clear. Now, if EC alone does not mean a lot, how to classify an EC system? It depends on the exact implementation of what happens when there is a partition, and what happens when the partition heals. There are EC systems that are configured by default to accept writes in the minority side (even in partitions when there is a single DB node if we want to honor "A" of CAP), and later will merge depending on "wall clock" timestamps. This is a form of consistency (and is weak), but is not Consistency with capital-C of CAP. Other systems will use vector clocks instead of timestamps, which is stronger but slower. Other systems will instead merge values together doing a "Set union" operation. Here we still have no strong consistency, but the safety of the system is improved. Now that we understand what means strong consistency, and what means eventual consistency, we can talk about Redis Cluster from this point of view. In Redis Cluster, which is eventual consistent, when the partition heals all the nodes will agree by selecting an history between the histories available for a given "hash slot" (a set of keys). However we need to consider what happens during partitions to have a better picture of what happens. In the case of Redis Cluster the minority side of the partition will stop accepting writes after some time. This gives up "A" of CAP in order to put a bound to how much two histories can diverge. Ok, now let's explore "A". CAP Availability is the ability to reply with a positive reply (so, basically, to be able to accept writes is required) to requests even in a minority partition with a single node. This is a very strong idea of availability, but this is not the only possibility to be fault tolerant. For example CP systems don't feature "A", but they are able to work correctly even if N/2-1 nodes will fail. This is a pretty strong failure tolerance! So they are reliable systems for a lot of works even if they don't have "A". As a counter example, Cassandra or Riak would still be very useful tools even without "A" availability, but just with N/2-1 nodes allowed to fail. Ok, back to Redis Cluster. It does not provides "A" since minority partitions are not available. It does not provide "C" since it is eventually consistent. But it still provides a form of consistency (its contract with the user about what happens during partitions) and provides some degree of resilience to failures. AP or CP are just the theoretical limits you can get if you want to exploit the max in this regard, however it is not like a system is useless if it is not AP or CP. In the case of Redis Cluster to renounce to "C" was obvious since it is asynchronously replicated and is not a good fit for high latency interactions. To renounce to the strong availability of "A" was a tradeoff to feature Redis data structures that are limited in size (number of elements) only by available memory. Suggested reading about EC: http://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is-not-safe/ http://www.bailis.org/blog/safety-and-liveness-eventual-cons...
lucidrains 13y ago
Thank you so so much for being in this world and creating Redis, it is wonderful!
miscoowow 13y ago
Since I switched to Redis, I had massive drop in resource requirements, saving me a lot of money. Thank you very much. The keyspace notifications are great but I use NodeJS at the moment to listen to the events. When will it be possible to run Lua to remove the current point of failure? On the topics page, it says "In the future". Are we looking at years or months?
- antirez 13y ago
  Awesome that you improved your stack with Redis, thanks for sharing. I'm not sure about what exact feature you are referring to, please could you describe it? Thanks.
  - miscoowow 13y ago
    http://redis.io/topics/notifications http://redis.io/topics/notifications notify-keyspace-events Kx Currently, NodeJS subscribes to expired keys. When the key is expired it will delete the 'physical' data from the server (thumbnails, more indepth info, etc). If NodeJS fails, even for a few seconds, I will forever have the 'physical' data stored on the server (as it doesn't re-run the expired event). Having lua scripting do what NodeJS currently does would fix the problem. Apart from that, Redis is perfect. ;)
    - seiji 13y ago
      I experimented with adding exactly that recently: https://matt.sh/advanced-redis-pubsub-scripts https://matt.sh/advanced-redis-pubsub-scripts (code in the branch at https://github.com/mattsta/redis/tree/pubsub-scripting https://github.com/mattsta/redis/tree/pubsub-scripting). It hasn't been extensively tested, but it works for my isolated test cases.
diminoten 13y ago
The most mundane question in the world - why not 2.9? New feature usually means minor version increment, yes?
- antirez 13y ago
  In order to simplify the development process I've this strange "rule" that new things are admissible in the stable branch patch releases as long as they are very very unlikely to interfere with all the rest. For example in this case, a testing environment and the other Sentinel changes actually improve reliability. From the point of view of BITPOS, it is a new command that if not used should not modify the behaviour/stability of the rest of the system in any way. However compare this process with creating a new release. With 2.8.7 I just need to write a changelog, tag the release on Github, and update a few numbers in the web site. With a release the process is very different, we have a number of release candidates before switching to stable and so forth.

by antirez 13y ago

ksec 13y ago

Just want to say, Thank You for Redis.

antirez 13y ago
Thank you ksec, without a great community there is no fun.
Jakehp 13y ago
+1. I just started learning and using it a few weeks ago; it's been a lot of fun.

leobelle 13y ago

Thanks for answering questions, any comments or progress on aphyr's complaints about using proven algorithms for Redis replication? [1] [1] http://aphyr.com/posts/307-call-me-maybe-redis-redux http://aphyr.com/posts/307-call-me-maybe-redis-redux

antirez 13y ago
Given that Apyhr's investigation in the blog post you cite does not involve anything implemented or planned for Redis, I'm not able to reply to the question. Maybe you fell in the mediatic trap of considering that discussion about Redis turning into a CP system using the proposed algorithm. If you re-read the post, Aphyr itself explains that this is not Redis Cluster. If you are curious about what triggered the post, is my claim that WAIT per se as a primitive does not mean CP or any other kind of consistency, but that it is defined by the rest of the system. To prove my point, I orchestrated a toy and god-driven distributed system where WAIT turned into a strong consistent primitive. The point is that you can replace the "God" part with other stuff to get it real-world. Apyhr modeled it and found that if you don't read the same way you write (via agreement) the system is inconsistent when you consider reads, but I think this was common knowledge. Every CP system in the base case needs to follow the same path for writes and reads. However most of the times you can invent optimizations to speedup reads. p.s. Redis replication is asynchronous by default. The WAIT primitive in Redis unstable does just what it claims, that is, to return an acknowledgment when the specified number of replicas received some data. The practical effect is that you can make writes safer for a (big) latency cost.
- lucian1900 13y ago
  You answered a different question. Are you using a verified consensus algorithm? If not, why are you making a distributed database? You're not inspiring confidence in using anything but single node Redis.
  - redbad 13y ago
    It seems that antirez believes he's allowing Redis to approach AP and/or CP semantics by providing primitives that users can compose in ad-hoc ways according to their requirements. Unfortunately, that's not a valid methodology to achieve availability or (especially) consistency.
    - antirez 13y ago
      In Redis Cluster there is nothing you can do to tune availability or consistency. The only thing you can tune is write "safety" using WAIT, but actually most users should never use it since the system is mostly designed to be low-latency. Redis Cluster is not AP nor CP. It resembles more AP since it has no strong consistency, however it is not able to reach "A" of AP that is very strict. Like AP systems have some form of consistency, but not "C" of CAP. Redis does this for "A" as well, it has some form of availability, but not as strict as "A" of CAP. Why this? You'll notice that there are not AP database systems with the powerful commands of Redis, and that there are not CP systems with the latency of Redis. So it is a very opinionated trade off to get both sacrificing other stuff (especially "A").
      - leobelle 13y ago
        If it's not AP or CP, then what is the value? Cassandra is eventually consistent. Consistency is pretty important. It seems like Redis Cluster doesn't have consistency, nor availability, nor partition tolerance. Maybe I misunderstand it though.
        antirez 13y ago
        > Cassandra is eventually consistent. Consistency is pretty important. Your above statement unfortunately has no match in distributed systems theory or practice. Basically eventually consistent means that the system is not consistent from the point of view of CAP, so it does not feature strong consistency. As an alternative eventual consistency means: our contract with the user is, that, at least, when there are no partitions, the system will no show split-brain conditions, so a given information will converge to a given value for all the nodes. This alone is a contract with the user, but does not tell a lot, it just tells you: 1) No strong consistency, sorry. 2) No split-brain conditions when the partitions heal. So "It features eventual consistency, consistency is important" statement, I'm afraid, means that you don't have the tradeoffs about distributed systems clear. Now, if EC alone does not mean a lot, how to classify an EC system? It depends on the exact implementation of what happens when there is a partition, and what happens when the partition heals. There are EC systems that are configured by default to accept writes in the minority side (even in partitions when there is a single DB node if we want to honor "A" of CAP), and later will merge depending on "wall clock" timestamps. This is a form of consistency (and is weak), but is not Consistency with capital-C of CAP. Other systems will use vector clocks instead of timestamps, which is stronger but slower. Other systems will instead merge values together doing a "Set union" operation. Here we still have no strong consistency, but the safety of the system is improved. Now that we understand what means strong consistency, and what means eventual consistency, we can talk about Redis Cluster from this point of view. In Redis Cluster, which is eventual consistent, when the partition heals all the nodes will agree by selecting an history between the histories available for a given "hash slot" (a set of keys). However we need to consider what happens during partitions to have a better picture of what happens. In the case of Redis Cluster the minority side of the partition will stop accepting writes after some time. This gives up "A" of CAP in order to put a bound to how much two histories can diverge. Ok, now let's explore "A". CAP Availability is the ability to reply with a positive reply (so, basically, to be able to accept writes is required) to requests even in a minority partition with a single node. This is a very strong idea of availability, but this is not the only possibility to be fault tolerant. For example CP systems don't feature "A", but they are able to work correctly even if N/2-1 nodes will fail. This is a pretty strong failure tolerance! So they are reliable systems for a lot of works even if they don't have "A". As a counter example, Cassandra or Riak would still be very useful tools even without "A" availability, but just with N/2-1 nodes allowed to fail. Ok, back to Redis Cluster. It does not provides "A" since minority partitions are not available. It does not provide "C" since it is eventually consistent. But it still provides a form of consistency (its contract with the user about what happens during partitions) and provides some degree of resilience to failures. AP or CP are just the theoretical limits you can get if you want to exploit the max in this regard, however it is not like a system is useless if it is not AP or CP. In the case of Redis Cluster to renounce to "C" was obvious since it is asynchronously replicated and is not a good fit for high latency interactions. To renounce to the strong availability of "A" was a tradeoff to feature Redis data structures that are limited in size (number of elements) only by available memory. Suggested reading about EC: http://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is-not-safe/ http://www.bailis.org/blog/safety-and-liveness-eventual-cons...

lucidrains 13y ago

Thank you so so much for being in this world and creating Redis, it is wonderful!

miscoowow 13y ago

Since I switched to Redis, I had massive drop in resource requirements, saving me a lot of money. Thank you very much. The keyspace notifications are great but I use NodeJS at the moment to listen to the events. When will it be possible to run Lua to remove the current point of failure? On the topics page, it says "In the future". Are we looking at years or months?

antirez 13y ago
Awesome that you improved your stack with Redis, thanks for sharing. I'm not sure about what exact feature you are referring to, please could you describe it? Thanks.
- miscoowow 13y ago
  http://redis.io/topics/notifications http://redis.io/topics/notifications notify-keyspace-events Kx Currently, NodeJS subscribes to expired keys. When the key is expired it will delete the 'physical' data from the server (thumbnails, more indepth info, etc). If NodeJS fails, even for a few seconds, I will forever have the 'physical' data stored on the server (as it doesn't re-run the expired event). Having lua scripting do what NodeJS currently does would fix the problem. Apart from that, Redis is perfect. ;)
  - seiji 13y ago
    I experimented with adding exactly that recently: https://matt.sh/advanced-redis-pubsub-scripts https://matt.sh/advanced-redis-pubsub-scripts (code in the branch at https://github.com/mattsta/redis/tree/pubsub-scripting https://github.com/mattsta/redis/tree/pubsub-scripting). It hasn't been extensively tested, but it works for my isolated test cases.

diminoten 13y ago

The most mundane question in the world - why not 2.9? New feature usually means minor version increment, yes?

antirez 13y ago
In order to simplify the development process I've this strange "rule" that new things are admissible in the stable branch patch releases as long as they are very very unlikely to interfere with all the rest. For example in this case, a testing environment and the other Sentinel changes actually improve reliability. From the point of view of BITPOS, it is a new command that if not used should not modify the behaviour/stability of the rest of the system in any way. However compare this process with creating a new release. With 2.8.7 I just need to write a changelog, tag the release on Github, and update a few numbers in the web site. With a release the process is very different, we have a number of release candidates before switching to stable and so forth.