5 ms·

Redis 5.0 RC1 is out

166 points by dsr12 8y ago

koolba 8y ago
Streams (https://redis.io/topics/streams-intro https://redis.io/topics/streams-intro) are going to be awesome. Blocking sorted set pops will also be quite useful. Looking forward to this being widely deployed enough to start using them. I don't see TLS support in the release notes though I remember antirez mentioning reviewing a PR for it. Is that being handled separately and possibly back patched?
- antirez 8y ago
  Unfortunately there was no way to put TLS so late in the game. We tried, but after I and my colleague Fabio reviewed the code, it was clear that in order to be a complete implementation the TLS support hooked and modified many things in the Redis internals, including sensitive things like replication. We'll try during Redis 6, now that we have time to do the things in a proper way.
  - koolba 8y ago
    That sounds more than reasonable as "measure twice, cut once" is the right answer for something as critical as a data store. I've said it before and I'll say it again, thanks for Redis!
    - antirez 8y ago
      Thanks! Btw I really hope we'll be able to go forward with the TLS patch ASAP. For the same reasons as above it will be great if we could merge/adapt it in the early stage of Redis 6, so that there will be a lot of time to test it.
- gaius 8y ago
  Once Streams are Prod ready, no more need for Kafka and hence Zookeeper as well.
kzrdude 8y ago
Side track, how is the Windows port of redis doing? Is it likely to be updated?
- antirez 8y ago
  AFAIK Microsoft abandoned the Win port, and AFAIK they run the Linux version on Azure. However with Windows 10 Linux subsystem you can run Redis for development without issues, just "make" and it's going to work out of the box.
  - breakingcups 8y ago
    Do you know why they abandoned it? Or why they started it?
    - antirez 8y ago
      They started the port after seeing I had no interest in writing a port for Windows. They stopped it IMHO when they realized, basically, the same things that prevented me from starting the project at all, that is: it runs very well on Linux and is very hard to port to Windows in a serious way, because of Windows lack of fork(). If you ask me Windows should absolutely get a serios POSIX layer ASAP and just let the Windows users use all the OSS that is out there.
      - WorldMaker 8y ago
        Which is what the Windows Subsystem for Linux does. If I recall correctly, Azure uses some combination of Windows Docker container and WSL support now to run Redis on Windows. Though from what I heard (as scuttlebutt/rumor/lay person reading the tea leaves), that isn't why they abandoned the port (as native ports are still handy for performance, and encouraged by Microsoft as much as they can), I heard it had more to do with the usual OSS reason of abandoning a port: not enough upstream support/enthusiasm for merging necessary, underlying patches, and not enough time/energy to devote to keeping a fork up-to-date with a wildly diverging upstream in such a situation. (Compare Git for Windows or Node for Windows, which have been officially recognized upstream, and directly merge a lot of patches upstream from Microsoft's contributions.) I don't have a dog in the fight and think you (antirez) are doing a good job, and Microsoft independently are doing a good job with OSS the last few years, and while I'm disappointed there wasn't more Reese's Cup peanut-butter-and-chocolate collaboration here between y'all, I'm okay eating my peanut butter and chocolate separately. That's OSS, in all its bazaar beauty.
antirez 8y ago
This Twitter thread contains more information / pointers: https://twitter.com/antirez/status/1001480569442783240 https://twitter.com/antirez/status/1001480569442783240
- jazoom 8y ago
  >Yes I totally appreciated your Tweet indeed, don’t get me wrong. I was just trying to communicate that in general I try to ignore stupid rules even if they are pretty damn real in the industry. Thanks! Right on!
alberth 8y ago
Has the Redis codebase stabilized? I mean that with no disrespect but the 4.x branch seemed to have a number of CRITICAL bug https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEA... Again, many thanks to Antirez who’s mostly a one-man-show.
- antirez 8y ago
  Hello alberth, I would not say Redis has an history of being unstable, actually it is regarded as not failing and not crashing often. However at the same time, some people believe that it is operationally challenging, which is a different problem. That Redis operations are harder than the ones of other DBs is in part true because the fact of having the data in memory, with different mechanisms to backup the data on disk, means that in case of an operation error (not necessarily a bug, just also an admin error), the outcomes can be worse. However in the filed of operations Redis improved year after year. However this does not answer your question about critical bugs. Basically most of the critical bugs you'll see in the Redis 4.0 changelog is about the new version of replication that was put in Redis in Redis 4.0 itself. The new protocol, called PSYNC2, was harder to implement and has many corner cases because it is able to restore the replication link with the master after it was broken. To do so requires to take the state of the replication intact, and given that slaves can do that (partial resynchronization) after an RDB restart, we found many issues in this respect: after an RDB restart there were cases where the state was not fully restored. Most of the time it was very hard to trigger such bugs, and in most cases those issues were found analytically and never reported. Yet I feel like I should mark every of such bug as CRITICAL in the release, regardless of the fact nobody was affected for a log time.
  - chandanrai 8y ago
    Wow, humbleness embodied! As great is redis,so is the maker.
didip 8y ago
redis-trib.rb is finally ported to redis-cli. Thanks Salvatore! Stream data type looks very interesting as well. Is the goal to be competitive to Kafka-like use-case?
- antirez 8y ago
  Hello didip, I've a non-competitive style when the matter is software :-) I simply think that Redis streams could be very useful, in some way I borrowed Kafka ideas, in other ways I totally refused the Kafka implementation / approach, and in general Redis streams have a lot of cons and pro at the same time in a mixed way, due to the total difference between Kafka end Redis. So I think it's the users that should apply the right tools for the right jobs, I'll avoid any comparison. I just gave Kafka credits in the Redis doc because certain ideas are clearly from Kafka or popularized by Kafka.
  - lossolo 8y ago
    Hello Salvatore, in which cases you would not use Redis streams and used Kafka/other solutions ? And in which cases you think Redis streams shine compared to other solutions ?
    - antirez 8y ago
      Hello lossolo, I hope that the community will figure this out. I'll try to make videos, blog posts and programming examples about Streams in Redis, but I don't like to make direct comparisons and I don't want to become a Kafka/other solution expert as well in order to make sure to make fair comparison :-) If I think at Redis Streams, I think they work very well in a set of domains. The king use case IMHO is time series. Having fast range queries is pretty interesting for this use case. Then the second one is events and event sourcing of all the kinds when the number of events-per-second is very sustained. However I've some feeling that also thinks like delayed jobs that are currently implemented on top of blocking lists may be rewritten to use streams and offer better overall features/operations.
      - 8y ago
ralusek 8y ago
I made a similar comment when streams were first announced, but didn't get feedback. My question is about using streams for event-sourcing, in similar cases to how Kafka is frequently used. In such cases, the expectation for persistence is paramount, as the event log is ultimately the primary source of truth for the entire application. The actual utility of having the event log in memory is only really important while the consumer groups are reading it, after which it mostly just needs to be archived. So my question is, are the common strategies used by Redis to persist to disk going to try to keep the whole event log in memory, or will they be able to intelligently offload to disk? Otherwise, is the expectation that clients will instead write to the stream, and consumers will be responsible for consuming, writing to some persistent layer like s3, and then removing from the stream?
- antirez 8y ago
  Hello ralusek, the idea is to provide more the tools for the application layer to move data from/to stream. I don't think we'll have direct support for persisting streams on disk. Honestly streams as implemented in Redis are very adapt to this use case, because of the macro-nodes that are swappable as a single entity, however some time ago I decided that I wanted to focus on the on-memory aspect.
  - ralusek 8y ago
    Cool, thank you for response.
erulabs 8y ago
Congrats & thanks Salvatore! Redis Streams is an awesome feature, and bringing Redis Cluster management tools into redis-cli is a world of improvement! Keep up the great work, hope to see you at next RedisConf!
- antirez 8y ago
  Thanks! See you soon, there will be a Redis event in London btw in case anyone is interested. Should be November 2018 AFAIK.
  - atomi 8y ago
    Will you be providing docker images for the rc releases? https://hub.docker.com/_/redis/ https://hub.docker.com/_/redis/
    - antirez 8y ago
      I'll ask my colleagues at Redis Labs if they can do it. Thanks for the hint.
      - omginternets 8y ago
        I would like to second this request. Thank you very much for your responsiveness :)
nasmorn 8y ago
I have a bad time with redis hosting from time to time. It is surely my own fault. What are your favorite resources/recipes to configure/host redis on your own?
- esterly 8y ago
  We've found Redis installs with no issues see: https://redis.io/topics/quickstart https://redis.io/topics/quickstart or if you are on windows https://github.com/rgl/redis/downloads https://github.com/rgl/redis/downloads On Ubuntu this guide works well https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-redis-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-inst... Only issue I've had is remembering to add the port to iptables
  - antirez 8y ago
    Thanks, a few other tips are here: https://redis.io/topics/admin https://redis.io/topics/admin https://redis.io/topics/latency https://redis.io/topics/latency In general our operations documentation does not still reflect all the good things put into Redis 4, we'll update everything shortly.
dstroot 8y ago
#1 - Congratulations! #2 - I am interested more in where clustering is going and running Redis inside K8S. Is there something someone can point me to to read on best practices for Redis 5?

166 points by dsr12 8y ago

Streams (https://redis.io/topics/streams-intro https://redis.io/topics/streams-intro) are going to be awesome. Blocking sorted set pops will also be quite useful. Looking forward to this being widely deployed enough to start using them. I don't see TLS support in the release notes though I remember antirez mentioning reviewing a PR for it. Is that being handled separately and possibly back patched?

antirez 8y ago
Unfortunately there was no way to put TLS so late in the game. We tried, but after I and my colleague Fabio reviewed the code, it was clear that in order to be a complete implementation the TLS support hooked and modified many things in the Redis internals, including sensitive things like replication. We'll try during Redis 6, now that we have time to do the things in a proper way.
- koolba 8y ago
  That sounds more than reasonable as "measure twice, cut once" is the right answer for something as critical as a data store. I've said it before and I'll say it again, thanks for Redis!
  - antirez 8y ago
    Thanks! Btw I really hope we'll be able to go forward with the TLS patch ASAP. For the same reasons as above it will be great if we could merge/adapt it in the early stage of Redis 6, so that there will be a lot of time to test it.
gaius 8y ago
Once Streams are Prod ready, no more need for Kafka and hence Zookeeper as well.

kzrdude 8y ago

Side track, how is the Windows port of redis doing? Is it likely to be updated?

antirez 8y ago
AFAIK Microsoft abandoned the Win port, and AFAIK they run the Linux version on Azure. However with Windows 10 Linux subsystem you can run Redis for development without issues, just "make" and it's going to work out of the box.
- breakingcups 8y ago
  Do you know why they abandoned it? Or why they started it?
  - antirez 8y ago
    They started the port after seeing I had no interest in writing a port for Windows. They stopped it IMHO when they realized, basically, the same things that prevented me from starting the project at all, that is: it runs very well on Linux and is very hard to port to Windows in a serious way, because of Windows lack of fork(). If you ask me Windows should absolutely get a serios POSIX layer ASAP and just let the Windows users use all the OSS that is out there.
    - WorldMaker 8y ago
      Which is what the Windows Subsystem for Linux does. If I recall correctly, Azure uses some combination of Windows Docker container and WSL support now to run Redis on Windows. Though from what I heard (as scuttlebutt/rumor/lay person reading the tea leaves), that isn't why they abandoned the port (as native ports are still handy for performance, and encouraged by Microsoft as much as they can), I heard it had more to do with the usual OSS reason of abandoning a port: not enough upstream support/enthusiasm for merging necessary, underlying patches, and not enough time/energy to devote to keeping a fork up-to-date with a wildly diverging upstream in such a situation. (Compare Git for Windows or Node for Windows, which have been officially recognized upstream, and directly merge a lot of patches upstream from Microsoft's contributions.) I don't have a dog in the fight and think you (antirez) are doing a good job, and Microsoft independently are doing a good job with OSS the last few years, and while I'm disappointed there wasn't more Reese's Cup peanut-butter-and-chocolate collaboration here between y'all, I'm okay eating my peanut butter and chocolate separately. That's OSS, in all its bazaar beauty.

antirez 8y ago

This Twitter thread contains more information / pointers: https://twitter.com/antirez/status/1001480569442783240 https://twitter.com/antirez/status/1001480569442783240

jazoom 8y ago
>Yes I totally appreciated your Tweet indeed, don’t get me wrong. I was just trying to communicate that in general I try to ignore stupid rules even if they are pretty damn real in the industry. Thanks! Right on!

alberth 8y ago

Has the Redis codebase stabilized? I mean that with no disrespect but the 4.x branch seemed to have a number of CRITICAL bug https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEA... Again, many thanks to Antirez who’s mostly a one-man-show.

antirez 8y ago
Hello alberth, I would not say Redis has an history of being unstable, actually it is regarded as not failing and not crashing often. However at the same time, some people believe that it is operationally challenging, which is a different problem. That Redis operations are harder than the ones of other DBs is in part true because the fact of having the data in memory, with different mechanisms to backup the data on disk, means that in case of an operation error (not necessarily a bug, just also an admin error), the outcomes can be worse. However in the filed of operations Redis improved year after year. However this does not answer your question about critical bugs. Basically most of the critical bugs you'll see in the Redis 4.0 changelog is about the new version of replication that was put in Redis in Redis 4.0 itself. The new protocol, called PSYNC2, was harder to implement and has many corner cases because it is able to restore the replication link with the master after it was broken. To do so requires to take the state of the replication intact, and given that slaves can do that (partial resynchronization) after an RDB restart, we found many issues in this respect: after an RDB restart there were cases where the state was not fully restored. Most of the time it was very hard to trigger such bugs, and in most cases those issues were found analytically and never reported. Yet I feel like I should mark every of such bug as CRITICAL in the release, regardless of the fact nobody was affected for a log time.
- chandanrai 8y ago
  Wow, humbleness embodied! As great is redis,so is the maker.

didip 8y ago

redis-trib.rb is finally ported to redis-cli. Thanks Salvatore! Stream data type looks very interesting as well. Is the goal to be competitive to Kafka-like use-case?

antirez 8y ago
Hello didip, I've a non-competitive style when the matter is software :-) I simply think that Redis streams could be very useful, in some way I borrowed Kafka ideas, in other ways I totally refused the Kafka implementation / approach, and in general Redis streams have a lot of cons and pro at the same time in a mixed way, due to the total difference between Kafka end Redis. So I think it's the users that should apply the right tools for the right jobs, I'll avoid any comparison. I just gave Kafka credits in the Redis doc because certain ideas are clearly from Kafka or popularized by Kafka.
- lossolo 8y ago
  Hello Salvatore, in which cases you would not use Redis streams and used Kafka/other solutions ? And in which cases you think Redis streams shine compared to other solutions ?
  - antirez 8y ago
    Hello lossolo, I hope that the community will figure this out. I'll try to make videos, blog posts and programming examples about Streams in Redis, but I don't like to make direct comparisons and I don't want to become a Kafka/other solution expert as well in order to make sure to make fair comparison :-) If I think at Redis Streams, I think they work very well in a set of domains. The king use case IMHO is time series. Having fast range queries is pretty interesting for this use case. Then the second one is events and event sourcing of all the kinds when the number of events-per-second is very sustained. However I've some feeling that also thinks like delayed jobs that are currently implemented on top of blocking lists may be rewritten to use streams and offer better overall features/operations.
    - 8y ago

ralusek 8y ago

I made a similar comment when streams were first announced, but didn't get feedback. My question is about using streams for event-sourcing, in similar cases to how Kafka is frequently used. In such cases, the expectation for persistence is paramount, as the event log is ultimately the primary source of truth for the entire application. The actual utility of having the event log in memory is only really important while the consumer groups are reading it, after which it mostly just needs to be archived. So my question is, are the common strategies used by Redis to persist to disk going to try to keep the whole event log in memory, or will they be able to intelligently offload to disk? Otherwise, is the expectation that clients will instead write to the stream, and consumers will be responsible for consuming, writing to some persistent layer like s3, and then removing from the stream?

antirez 8y ago
Hello ralusek, the idea is to provide more the tools for the application layer to move data from/to stream. I don't think we'll have direct support for persisting streams on disk. Honestly streams as implemented in Redis are very adapt to this use case, because of the macro-nodes that are swappable as a single entity, however some time ago I decided that I wanted to focus on the on-memory aspect.
- ralusek 8y ago
  Cool, thank you for response.

erulabs 8y ago

Congrats & thanks Salvatore! Redis Streams is an awesome feature, and bringing Redis Cluster management tools into redis-cli is a world of improvement! Keep up the great work, hope to see you at next RedisConf!

antirez 8y ago
Thanks! See you soon, there will be a Redis event in London btw in case anyone is interested. Should be November 2018 AFAIK.
- atomi 8y ago
  Will you be providing docker images for the rc releases? https://hub.docker.com/_/redis/ https://hub.docker.com/_/redis/
  - antirez 8y ago
    I'll ask my colleagues at Redis Labs if they can do it. Thanks for the hint.
    - omginternets 8y ago
      I would like to second this request. Thank you very much for your responsiveness :)

nasmorn 8y ago

I have a bad time with redis hosting from time to time. It is surely my own fault. What are your favorite resources/recipes to configure/host redis on your own?

esterly 8y ago
We've found Redis installs with no issues see: https://redis.io/topics/quickstart https://redis.io/topics/quickstart or if you are on windows https://github.com/rgl/redis/downloads https://github.com/rgl/redis/downloads On Ubuntu this guide works well https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-redis-on-ubuntu-16-04 https://www.digitalocean.com/community/tutorials/how-to-inst... Only issue I've had is remembering to add the port to iptables
- antirez 8y ago
  Thanks, a few other tips are here: https://redis.io/topics/admin https://redis.io/topics/admin https://redis.io/topics/latency https://redis.io/topics/latency In general our operations documentation does not still reflect all the good things put into Redis 4, we'll update everything shortly.

dstroot 8y ago

#1 - Congratulations! #2 - I am interested more in where clustering is going and running Redis inside K8S. Is there something someone can point me to to read on best practices for Redis 5?