20 ms·

Contrarian view here. I really liked Redis for a long time. Simple, fast data-structures in-memory. That's it. Along the way there have been some nice enhancem

by _pgmf 6y ago

Contrarian view here. I really liked Redis for a long time. Simple, fast data-structures in-memory. That's it. Along the way there have been some nice enhancements like Lua, which solves a lot of the atomicity issues. But somewhere after 4.0 I feel they have lost their way. First they saw all the attention Kafka/event-stuff was receiving, so they baked-in this monstrous, complicated, stateful streams feature. Now we have SSL (do people really expose Redis on the internet??), ACLs, cluster stuff, and most relevant to me a new wire protocol. To my thinking, Redis fit very well in the "lightweight linux thing" category. It seems they aspire to be enterprise software, and this may be a good move for Redis-the-Business, but it's not good for users like me who just want simple in-memory data-structures and as little state as possible. Forcing a new protocol that adds very little value (in my opinion) also seems like a great way to alienate your users.

brodouevencode 6y ago
Agree on the Kafka, disagree on the SSL. There are compliance factors in place for certain use cases (PII).
- sgt 6y ago
  As a comparison, we run (and I think this is pretty common) nginx proxy servers that point to app servers. The proxy servers handle SSL to the outside, whilst the connection to the app servers is simply http. Pretty sure that is an acceptable solution in most cases. So then this would apply to the SSL argument here as well.
  - jrockway 6y ago
    You can stick a proxy in front of apps that don't have features you need like mTLS, tracing, metrics, etc. to get those. Google "service mesh" to explore that space. But to some extent, I think it's all a bit easier if your apps just do the right thing out of the box. Less moving parts. Better integration testing. Like medicine, every piece of software you use has effects and side-effects. If the advantage of the effects outweigh the disadvantages of the side-effects, then something is a good deal. But if you can avoid the side-effects entirely, that's best.
  - pritambarhate 6y ago
    >> Pretty sure that is an acceptable solution in most cases No, it's not. In almost all high-security compliance audits you are required to have the data encrypted anytime it's in-flight.
    - brodouevencode 6y ago
      Some of our systems require in-flight and at rest encryption.
  - singlow 6y ago
    All network traffic that leaves a host should be encrypted. You could have an exception for a physically isolated network in a secure cage, if you're adventurous. But most of use are in cloud environments, so encrypted traffic is required. Even with VPCs and Security Groups, you don't want to rely on network ACLs alone to prevent data from being intercepted. If Redis does not support encryption natively, then you have to run a gateway like stunnel on every redis host. The redis clients mostly all already support connecting to a secure socket, but the server and cli client require manual stunnel configurations. Native support for encryption just removes this extra setup.
    - vetinari 6y ago
      Encrypted doesn't mean TLS necessarily, and neither it means that it has to be at layer 7. For the use case you are suggesting, wireguard network among your hosts would be simpler and without the need for each application to handle TLS.
      - icebraining 6y ago
        And how would we then connect our section of Windows workers, which are needed to run some proprietary software? As long as wireguard is Linux-only, it's not an universal solution. TLS is.
  - _jal 6y ago
    We did, too, when we were in startup mode. Now, nothing runs unencrypted internally. Most tooling uses TLS, because when you do this at scale, you automate your CA and it is much easier to securely deal with than, eg, ssh certs. But we do use (LDAP centralized) ssh as well, mostly for humans.
e12e 6y ago
> Now we have SSL (do people really expose Redis on the internet??) There are no secure networks. Your options are vpn, third party ssl, or ssl in the service. Sometimes, your datacenter/cloud will guarantee "secure" network (ie: manage vpn for you). But in many instances having ssl "inside" can be simpler. Postgres also offers secure transport.
- coleifer 6y ago
  I suppose I shouldn't be surprised at this level of pedantic cherry-picking, but yet... I still find myself bristling.
  - justinclift 6y ago
    It's a bit unclear what you're meaning? Reading over the comment you replied to, I was thinking similar myself. :/
- gregoriol 6y ago
  Why not simply use wireguard instead of each tool implementing its own ssl support?
  - paulryanrogers 6y ago
    WG is young, it probably didn't exist in a stable form when TLS took hold in most projects. It's possible it may dethrone TLS in the future
  - d4mi3n 6y ago
    I think this could be workable, but it probably depends a lot on context. One reason off the top of my head would be regulatory/compliance issues around how things are encrypted. wireguard is relatively new, and some certifications required to do business in specific industries (finance, healthcare, etc) mandate protocols with a minimum level of maturity. wireguard may be good, but many regulators would probably not find it acceptable without a longer track record. On a more concrete note, I'd consider any system that handles authentication to be inherently broken if it had no way to keep those credentials safe out of the box. TLS has long been a cheap-ish way to do this, as it's widely available and well understood by both implements and regulators.
d4mi3n 6y ago
Security guy here. I'd argue that SSL and ACL are always good things to have, especially for systems that store data. Modern security practices typically dictate a defense-in-depth approach. The ideas is that you will be compromised at some point (no security is perfect) and as such you should make any compromise that does happen as minimal as possible--you want to prevent attackers who get a foot in the door from rummaging around your network. A key part of any defense-in-depth strategy are things like encryption and authentication/authorization. If you're using redis to store any kind of sensitive material, you want to make sure that only people on your network with the appropriate auth credentials can access it. This is one of the easiest ways to prevent drive-by data theft. From here, SSL is a logical step. You need to ensure bad actors can't sniff network traffic and steal credentials. I can't speak to streams or the other features you feel complicate Redis, but I think SSL+ACL are very important tools for increasing the cost to attackers that target redis instances leveraging those features.
- oconnore 6y ago
  Many systems don’t do TLS in process. TLS proxying is probably more common for systems deployed in the cloud (e.g. running nginx on the same node, or using a cloud load balancer). AWS and GCP don’t even give you a way to install a cert yourself— you MUST use an ELB or bring your own certificate.
  - d4mi3n 6y ago
    This is highly dependent on your environment. I work in finance and there is legislation saying we must encrypt all traffic on the wire. Legislation aside, this also goes back to a defense-in-depth strategy; TLS proxying only works if the network behind the proxy will always be secure. You might be able to get away with running TLS on the same host as redis, but in all other cases I can think of you're going back to the 90's-era security policy of having a hard shell and a soft underbelly--anything that gets into the network behind your TLS proxy can sniff whatever traffic it wants. EDIT: It occurs to me that you seem to be hinting at running redis as a public service. In that scenario it makes perfect sense to use a TLS proxy for versions of redis without SSL. That said, it's still important to encrypt things on your private network to ensure you aren't one breach away from having your whole stack blown open.
antirez 6y ago
I understand the sentiment, but things are a bit different than they may look. About SSL, there is no way out of this. I opposed to this feature for a long time, but simply now because of changes in regulations, policies and so forth, a lot of use cases are migrating to SSL internally even if Redis is not exposed. And frankly it is really a mess to handle SSL proxies now that everybody look like needs encryption. So what I did was the best I could do, when checking for PRs to merge: 1) Opt in, not compiled by default, no SSL libs requirements. 2) Connection abstraction, there is no SSL mentioned inside the code. Everything is in a different file. About the "Kafka" thing, actually streams were wanted by myself, very strongly, and not suggested by Redis Labs. Let's start thinking at Redis ad a data structure server and at streams without the consumer groups part (which is totally optional). It was incredible we had no way to model a "log" in a serious way. No time series easily, using hacks and with huge memory usage because sorted sets are not the solution for this problem. But then why consumer groups? Because for a long time people had this problem of wanting a "persistent Pub/Sub": you can't lose messages just because clients disconnect in most use cases. Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice. But ACLs, in order to manage to survive 10 years without ACLs we had to resort to all kind of tricks: renaming commands to unguessable strings. Still with the panic of some library calling FLUSHALL for error because the developer was testing it in her/his laptop. Really ACLs have nothing to do with enterprise, but some safety is needed. The ACL monster is 1297 lines of code, and is one of the most user friendly security system you'll find in a database. Actually all those features have a great impact on the users, huge impact on day to day operations, and are designed in order to be as simple as possible. And Redis Labs actually has only to lose from all this, because those were all potential "premium" features, instead now they are in and every other Redis provider will have it automatically as a standard. So... reality is a bit different, and it's not a conspiracy to gain market shares or alike.
- subhobroto 6y ago
  Redis streams have been a phenomenal addition to my toolbelt in designing realtime ETL/ELT pipelines. Before I had to make do with a way more complicated Pub/Sub + job q (Tasktiger). That all became redundant thanks to Redis streams. Thank you! It would really be awesome if there was a built in way to attach/spill/persist individual streams to external data volumes (older/busy streams could run out of memory) and have it support hot swapping. > Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice. Hilarious, funny and informative :D Upvoted!
jrockway 6y ago
SSL is a pretty important feature for almost all apps that you run in the datacenter. The idea is not to securely send Redis data to an end user on an untrusted network, the idea is to reduce the blast radius of a compromise inside your datacenter. A good example is that Slack postmortem from a couple weeks ago -- they had a proxy running inside their datacenter, and it could be convinced to make connections to internal addresses. If the service it was trying to connect to required the client to have a valid TLS certificate, the proxy would likely not provide the right credentials (because who uses client certificates on the Internet), and the connection would simply fail. A big security bug would manifest as a higher error ration in the service, instead of letting an attacker poke around in their user data. (Network based policy is also good, but is often too broad a brush. You might want the proxy to be able to talk to a database server in your network to store some results; now you can't simply add a firewall rule that says "no traffic may pass to the internal network".) Finally, you might remember that internal NSA slideshow with the "SSL added and removed here ;-)" when talking about how they stole user data from Google's internal network. After that leak, rollout of internal mutual authentication/encryption accelerated, because people were actually inside the network observing internal RPC calls. It wasn't theoretical, it was happening. Ultimately, mTLS is a pretty simple way to get a large increase in security. It has its disadvantages; you now have another "moving part" that can interfere with deployments and debugging (an attacker can't tcpdump, and neither can you, easily), but given how many large companies have exposed user data through unintentional interactions, it is something worth considering. It's a technique where you have to go out of your way to cause surprising behavior, and that is always good for security and reliability.
gorkish 6y ago
I see Redis as a toolkit that collects a number of solutions to hard distributed system problems in a single tool. It is great for developers that have a number of use cases for these kinds of things but for which there is no need or justification to spool up yet another cluster of containers/vms/servers/load balancers/etc to support it. Redis already has to do these things to be reliable and consistent; directly exposing this ability to clients and modules is a very logical thing to do. Like it or not, Redis is a platform now. If you really just want fast data structures in memory, use memcached. If you somehow feel that Redis is a better solution for you, perhaps you should carefully consider that you may be placing more weight on its platform features than you realize.
time0ut 6y ago
Redis is painful to use in a highly regulated environment where all data must be encrypted in transit, all access logged and audited, etc. Personally, I think the requirements are over the top and focus on the wrong things a lot of the time. But it is what our compliance people say we must do. We've spent hundreds of hours cobbling together a system that meets our regulatory requirements and still performs well. These features go a long way toward addressing this pain point. I think they've done a decent job making the extra complexity optional, too.
kristoff_it 6y ago
I'm a dev advocate at Redis Labs so I'm just going to reply about the last point about the protocol from my developer PoV. Thanks to RESP3 I was able to write this client that, combined with Zig's comptime metaprogramming, can do things that no other Redis client that I've seen can do. The user gives the desired response type and the client is able to provide appropriate translation based on what the RESP3 reply is. This would still be possible with RESP2, but v3 makes it much more robust and explicit, to the point that the ease becomes transparent without looking magical and/or triggering confusing corner cases. https://github.com/kristoff-it/zig-okredis https://github.com/kristoff-it/zig-okredis
easytiger 6y ago
On SSL, with some corporate guidelines on moving to cloud providers you must prove comms between all hosts are encrypted
derefr 6y ago
> monstrous, complicated, stateful streams feature It's two data structures (which were already in Redis for other reasons!), and an automatic sequential identifier. Everything else that's "stateful" about it is client-side state—the server is still just a data-structure server. A Redis stream is basically just a Redis sorted set that's coherent in the face of clients trying to consume it paginated as other clients insert into the middle of it. Also, the code is in one file (https://github.com/antirez/redis/blob/unstable/src/t_stream.c https://github.com/antirez/redis/blob/unstable/src/t_stream.... ); that file is ~3KLOC. It's just another Redis Module, isolated into its own set of functions with no impact on the codebase as a whole. It's just one that's so widely applicable, to so many use-cases that people were already using Redis for (through Sidekiq/Resque/etc) that it makes sense to ship this particular module with Redis itself. Would you get upset about bloat if Postgres upstreamed a highly-popular extension? It already has nine or ten installed by default, and a few more sitting in contrib/. But, of course, even upstreamed, none of those extensions are enabled by default, adding runtime overhead to your DB; you have to ask for them, just as if you were installing a third-party extension. Same here: if you don't use the Streams module, there's no overhead to its existence in the Redis codebase. > do people really expose Redis on the internet?? Cloud DBaaS providers expose Redis instances "over the Internet", in the sense that they're in the same AZ but not within your VPC. To the extent that you can wireshark a data-center's virtual SDN, they need to encrypt this traffic. Even PaaS providers do things this way, since they usually lean on third-party DBaaS providers. E.g. all of the Redis services you can attach to a Heroku app are consumed "over the Internet." If you're using Redis through an IaaS provider's offering (e.g. AWS ElastiCache, Google Cloud Memorystore) then you get the benefit of them being able to spawn an instance "outside" your project/VPC (i.e. having it be managed by them), but have it nevertheless routed to an IP address inside your VPC. That might be enough security for you, if you don't have any legal requirements saying otherwise. For some people, it's not, and they need TLS on top anyway. > cluster stuff Have you looked at how it's done? It's just ease-of-use tooling around the obvious thing to do to scale Redis: partitioning the keyspace onto distinct Redis instances, and then routing requests to partitions based on the key. It's not like Redis has suddenly become a multi-master consensus system like Zookeeper; the router logic isn't even in the server codebase!
rockostrich 6y ago
> do people really expose Redis on the internet?? You've never browsed unprotected IPs and ports huh? So many random redis instances just lying around.
benmmurphy 6y ago
Having TLS support in the main client is useful because AWS only supports AUTH if you enable TLS. Running Redis without AUTH can be kind of dangerous because Redis can kind of speak HTTP* (I think you can define custom commands to fix this) so if you have web hooks in your system and don't properly filter internal addresses then you might allow external parties to run Redis commands against your system. * it's been years since I looked at this so maybe Redis now ships with inbuilt protection against this.
all_blue_chucks 6y ago
> do people really expose Redis on the internet? Do people really still send database traffic unencrypted over unencrypted internal networks?
stouset 6y ago
> Now we have SSL (do people really expose Redis on the internet??). This is 2020. The "hard outer shell, soft chewy center" model of security is dead and it's not coming back. Modern datacenters and cloud deployments use mTLS (mutually-authenticated TLS) between every service, everywhere, all the time. There are some massive benefits to this. For starters, you can limit what services talk to one-another entirely through distribution and signing of keys. Yes, this adds a burden of complexity if you go that route. But suddenly you don't have to care as much about (for instance) many network-exploitable vulnerabilities in your services because someone with a foothold on your network can't even get talk to your service in the first place if they don't have the right TLS cert, which is only on the handful of machines and only readable by the specific services that are legitimately allowed to connect to it. This is a much stronger guarantee than firewalling alone (though you should also use firewalling), because multiple services can be running on a host but only the applications that are allowed to talk to your service will have read access to that key. On the flip side, you have stronger guarantees that the service you're connecting to really is the service you're expecting it to be. If you're storing sensitive information in Redis, you can know for sure that the port you've connected to is the right Redis and not another, less-sensitive application's.
- notyourday 6y ago
  I hear this in marketing presentation or company pitches all the time. After that I as a consultant get access to the network and apart from some test that a developer stood up nothing matches the glossy talk. Thanks god for Wireguard. It has truly been the savior deploying encrypted networks.
  - jimmcslim 6y ago
    > After that I as a consultant get access to the network and apart from some test that a developer stood up nothing matches the glossy talk. Or in my case recently... someone has generated a root certificate for the internal CA that uses an insecure crypto scheme, and Chrome still throws up a security error requiring users to click past the warnings to access the site. "Can you generate and roll out a new cert please? This isn't really 'security'?" "Oh we will get to it, can you just use the one you already have?"
kawsper 6y ago
> do people really expose Redis on the internet?? Accidentally on a hobby-server, because docker-compose automatically opens up firewall ports.
Florin_Andrei 6y ago
Super-simple, good-enough things don't last too long. They die when the ecosystem changes, or when another super-simple player comes along and looks a bit more shiny. It's just how the world works. You have to conquer to survive.
vageli 6y ago
SSL/encrypted connections is a requirement in some regulatory frameworks. For example, transmitting PHI needs to be done over an encrypted connection.
apple4ever 6y ago
SSL/TLS is super important. I'm glad Redis added it. Now if only Varnish would get in the game.
manigandham 6y ago
If Redis is already good enough for you, what actually changes for you to start disliking it? You don't have to use any of those features.
redis_mlc 6y ago
> To my thinking, Redis fit very well in the "lightweight linux thing" category. It sounds like you don't follow Redis then. That ship sailed years ago. Redis has at least 10 major features in addition to the caching you're talking about, including search. Redis is a kind of database now. If you just want a cache, use memcached. Half of my jiras at one company were related to enabling SSL for Redis due to compliance reasons (all for internal use.) Now those can be closed.
dpipemazo 6y ago
We found streams to be a breakthrough feature for pub-sub type data on IoT devices. That it can both be low-latency pub-sub and a stateful, short-lived cache is quite powerful to improve performance for many queries to the types of data generated by cameras and high-frequency sensor devices. http://atomdocs.io/ http://atomdocs.io/ https://github.com/elementary-robotics/atom https://github.com/elementary-robotics/atom
__david__ 6y ago
Personally I'm jumping up and down for ACLs. I went so far as to implement a proof of concept Redis proxy that added ACLs a couple years ago, before I heard that they would be in 6. ACLs may be niche, but when you need 'em, you need 'em!
dagi3d 6y ago
The question would be if by adding those additional features, the experience to use the "basic/original" features got more cumbersome or the hardware requirements did change a lot. My guess it hasn't changed that much.
coleifer 6y ago
Replying to myself here. Of course every commenter wanted to bikeshed the SSL/crypto sentence. I forgot where I was commenting. I accidentally set the stage for a mountain of security circlejerking. No nibbles on the protocol point, which in fact was the main thing.

by _pgmf 6y ago

brodouevencode 6y ago

Agree on the Kafka, disagree on the SSL. There are compliance factors in place for certain use cases (PII).

sgt 6y ago
As a comparison, we run (and I think this is pretty common) nginx proxy servers that point to app servers. The proxy servers handle SSL to the outside, whilst the connection to the app servers is simply http. Pretty sure that is an acceptable solution in most cases. So then this would apply to the SSL argument here as well.
- jrockway 6y ago
  You can stick a proxy in front of apps that don't have features you need like mTLS, tracing, metrics, etc. to get those. Google "service mesh" to explore that space. But to some extent, I think it's all a bit easier if your apps just do the right thing out of the box. Less moving parts. Better integration testing. Like medicine, every piece of software you use has effects and side-effects. If the advantage of the effects outweigh the disadvantages of the side-effects, then something is a good deal. But if you can avoid the side-effects entirely, that's best.
- pritambarhate 6y ago
  >> Pretty sure that is an acceptable solution in most cases No, it's not. In almost all high-security compliance audits you are required to have the data encrypted anytime it's in-flight.
  - brodouevencode 6y ago
    Some of our systems require in-flight and at rest encryption.
- singlow 6y ago
  All network traffic that leaves a host should be encrypted. You could have an exception for a physically isolated network in a secure cage, if you're adventurous. But most of use are in cloud environments, so encrypted traffic is required. Even with VPCs and Security Groups, you don't want to rely on network ACLs alone to prevent data from being intercepted. If Redis does not support encryption natively, then you have to run a gateway like stunnel on every redis host. The redis clients mostly all already support connecting to a secure socket, but the server and cli client require manual stunnel configurations. Native support for encryption just removes this extra setup.
  - vetinari 6y ago
    Encrypted doesn't mean TLS necessarily, and neither it means that it has to be at layer 7. For the use case you are suggesting, wireguard network among your hosts would be simpler and without the need for each application to handle TLS.
    - icebraining 6y ago
      And how would we then connect our section of Windows workers, which are needed to run some proprietary software? As long as wireguard is Linux-only, it's not an universal solution. TLS is.
- _jal 6y ago
  We did, too, when we were in startup mode. Now, nothing runs unencrypted internally. Most tooling uses TLS, because when you do this at scale, you automate your CA and it is much easier to securely deal with than, eg, ssh certs. But we do use (LDAP centralized) ssh as well, mostly for humans.

e12e 6y ago

> Now we have SSL (do people really expose Redis on the internet??) There are no secure networks. Your options are vpn, third party ssl, or ssl in the service. Sometimes, your datacenter/cloud will guarantee "secure" network (ie: manage vpn for you). But in many instances having ssl "inside" can be simpler. Postgres also offers secure transport.

coleifer 6y ago
I suppose I shouldn't be surprised at this level of pedantic cherry-picking, but yet... I still find myself bristling.
- justinclift 6y ago
  It's a bit unclear what you're meaning? Reading over the comment you replied to, I was thinking similar myself. :/
gregoriol 6y ago
Why not simply use wireguard instead of each tool implementing its own ssl support?
- paulryanrogers 6y ago
  WG is young, it probably didn't exist in a stable form when TLS took hold in most projects. It's possible it may dethrone TLS in the future
- d4mi3n 6y ago
  I think this could be workable, but it probably depends a lot on context. One reason off the top of my head would be regulatory/compliance issues around how things are encrypted. wireguard is relatively new, and some certifications required to do business in specific industries (finance, healthcare, etc) mandate protocols with a minimum level of maturity. wireguard may be good, but many regulators would probably not find it acceptable without a longer track record. On a more concrete note, I'd consider any system that handles authentication to be inherently broken if it had no way to keep those credentials safe out of the box. TLS has long been a cheap-ish way to do this, as it's widely available and well understood by both implements and regulators.

d4mi3n 6y ago

Security guy here. I'd argue that SSL and ACL are always good things to have, especially for systems that store data. Modern security practices typically dictate a defense-in-depth approach. The ideas is that you will be compromised at some point (no security is perfect) and as such you should make any compromise that does happen as minimal as possible--you want to prevent attackers who get a foot in the door from rummaging around your network. A key part of any defense-in-depth strategy are things like encryption and authentication/authorization. If you're using redis to store any kind of sensitive material, you want to make sure that only people on your network with the appropriate auth credentials can access it. This is one of the easiest ways to prevent drive-by data theft. From here, SSL is a logical step. You need to ensure bad actors can't sniff network traffic and steal credentials. I can't speak to streams or the other features you feel complicate Redis, but I think SSL+ACL are very important tools for increasing the cost to attackers that target redis instances leveraging those features.

oconnore 6y ago
Many systems don’t do TLS in process. TLS proxying is probably more common for systems deployed in the cloud (e.g. running nginx on the same node, or using a cloud load balancer). AWS and GCP don’t even give you a way to install a cert yourself— you MUST use an ELB or bring your own certificate.
- d4mi3n 6y ago
  This is highly dependent on your environment. I work in finance and there is legislation saying we must encrypt all traffic on the wire. Legislation aside, this also goes back to a defense-in-depth strategy; TLS proxying only works if the network behind the proxy will always be secure. You might be able to get away with running TLS on the same host as redis, but in all other cases I can think of you're going back to the 90's-era security policy of having a hard shell and a soft underbelly--anything that gets into the network behind your TLS proxy can sniff whatever traffic it wants. EDIT: It occurs to me that you seem to be hinting at running redis as a public service. In that scenario it makes perfect sense to use a TLS proxy for versions of redis without SSL. That said, it's still important to encrypt things on your private network to ensure you aren't one breach away from having your whole stack blown open.

antirez 6y ago

I understand the sentiment, but things are a bit different than they may look. About SSL, there is no way out of this. I opposed to this feature for a long time, but simply now because of changes in regulations, policies and so forth, a lot of use cases are migrating to SSL internally even if Redis is not exposed. And frankly it is really a mess to handle SSL proxies now that everybody look like needs encryption. So what I did was the best I could do, when checking for PRs to merge: 1) Opt in, not compiled by default, no SSL libs requirements. 2) Connection abstraction, there is no SSL mentioned inside the code. Everything is in a different file. About the "Kafka" thing, actually streams were wanted by myself, very strongly, and not suggested by Redis Labs. Let's start thinking at Redis ad a data structure server and at streams without the consumer groups part (which is totally optional). It was incredible we had no way to model a "log" in a serious way. No time series easily, using hacks and with huge memory usage because sorted sets are not the solution for this problem. But then why consumer groups? Because for a long time people had this problem of wanting a "persistent Pub/Sub": you can't lose messages just because clients disconnect in most use cases. Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice. But ACLs, in order to manage to survive 10 years without ACLs we had to resort to all kind of tricks: renaming commands to unguessable strings. Still with the panic of some library calling FLUSHALL for error because the developer was testing it in her/his laptop. Really ACLs have nothing to do with enterprise, but some safety is needed. The ACL monster is 1297 lines of code, and is one of the most user friendly security system you'll find in a database. Actually all those features have a great impact on the users, huge impact on day to day operations, and are designed in order to be as simple as possible. And Redis Labs actually has only to lose from all this, because those were all potential "premium" features, instead now they are in and every other Redis provider will have it automatically as a standard. So... reality is a bit different, and it's not a conspiracy to gain market shares or alike.

subhobroto 6y ago
Redis streams have been a phenomenal addition to my toolbelt in designing realtime ETL/ELT pipelines. Before I had to make do with a way more complicated Pub/Sub + job q (Tasktiger). That all became redundant thanks to Redis streams. Thank you! It would really be awesome if there was a built in way to attach/spill/persist individual streams to external data volumes (older/busy streams could run out of memory) and have it support hot swapping. > Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice. Hilarious, funny and informative :D Upvoted!

jrockway 6y ago

SSL is a pretty important feature for almost all apps that you run in the datacenter. The idea is not to securely send Redis data to an end user on an untrusted network, the idea is to reduce the blast radius of a compromise inside your datacenter. A good example is that Slack postmortem from a couple weeks ago -- they had a proxy running inside their datacenter, and it could be convinced to make connections to internal addresses. If the service it was trying to connect to required the client to have a valid TLS certificate, the proxy would likely not provide the right credentials (because who uses client certificates on the Internet), and the connection would simply fail. A big security bug would manifest as a higher error ration in the service, instead of letting an attacker poke around in their user data. (Network based policy is also good, but is often too broad a brush. You might want the proxy to be able to talk to a database server in your network to store some results; now you can't simply add a firewall rule that says "no traffic may pass to the internal network".) Finally, you might remember that internal NSA slideshow with the "SSL added and removed here ;-)" when talking about how they stole user data from Google's internal network. After that leak, rollout of internal mutual authentication/encryption accelerated, because people were actually inside the network observing internal RPC calls. It wasn't theoretical, it was happening. Ultimately, mTLS is a pretty simple way to get a large increase in security. It has its disadvantages; you now have another "moving part" that can interfere with deployments and debugging (an attacker can't tcpdump, and neither can you, easily), but given how many large companies have exposed user data through unintentional interactions, it is something worth considering. It's a technique where you have to go out of your way to cause surprising behavior, and that is always good for security and reliability.

gorkish 6y ago

I see Redis as a toolkit that collects a number of solutions to hard distributed system problems in a single tool. It is great for developers that have a number of use cases for these kinds of things but for which there is no need or justification to spool up yet another cluster of containers/vms/servers/load balancers/etc to support it. Redis already has to do these things to be reliable and consistent; directly exposing this ability to clients and modules is a very logical thing to do. Like it or not, Redis is a platform now. If you really just want fast data structures in memory, use memcached. If you somehow feel that Redis is a better solution for you, perhaps you should carefully consider that you may be placing more weight on its platform features than you realize.

time0ut 6y ago

Redis is painful to use in a highly regulated environment where all data must be encrypted in transit, all access logged and audited, etc. Personally, I think the requirements are over the top and focus on the wrong things a lot of the time. But it is what our compliance people say we must do. We've spent hundreds of hours cobbling together a system that meets our regulatory requirements and still performs well. These features go a long way toward addressing this pain point. I think they've done a decent job making the extra complexity optional, too.

kristoff_it 6y ago

I'm a dev advocate at Redis Labs so I'm just going to reply about the last point about the protocol from my developer PoV. Thanks to RESP3 I was able to write this client that, combined with Zig's comptime metaprogramming, can do things that no other Redis client that I've seen can do. The user gives the desired response type and the client is able to provide appropriate translation based on what the RESP3 reply is. This would still be possible with RESP2, but v3 makes it much more robust and explicit, to the point that the ease becomes transparent without looking magical and/or triggering confusing corner cases. https://github.com/kristoff-it/zig-okredis https://github.com/kristoff-it/zig-okredis

easytiger 6y ago

On SSL, with some corporate guidelines on moving to cloud providers you must prove comms between all hosts are encrypted

derefr 6y ago

> monstrous, complicated, stateful streams feature It's two data structures (which were already in Redis for other reasons!), and an automatic sequential identifier. Everything else that's "stateful" about it is client-side state—the server is still just a data-structure server. A Redis stream is basically just a Redis sorted set that's coherent in the face of clients trying to consume it paginated as other clients insert into the middle of it. Also, the code is in one file (https://github.com/antirez/redis/blob/unstable/src/t_stream.c https://github.com/antirez/redis/blob/unstable/src/t_stream.... ); that file is ~3KLOC. It's just another Redis Module, isolated into its own set of functions with no impact on the codebase as a whole. It's just one that's so widely applicable, to so many use-cases that people were already using Redis for (through Sidekiq/Resque/etc) that it makes sense to ship this particular module with Redis itself. Would you get upset about bloat if Postgres upstreamed a highly-popular extension? It already has nine or ten installed by default, and a few more sitting in contrib/. But, of course, even upstreamed, none of those extensions are enabled by default, adding runtime overhead to your DB; you have to ask for them, just as if you were installing a third-party extension. Same here: if you don't use the Streams module, there's no overhead to its existence in the Redis codebase. > do people really expose Redis on the internet?? Cloud DBaaS providers expose Redis instances "over the Internet", in the sense that they're in the same AZ but not within your VPC. To the extent that you can wireshark a data-center's virtual SDN, they need to encrypt this traffic. Even PaaS providers do things this way, since they usually lean on third-party DBaaS providers. E.g. all of the Redis services you can attach to a Heroku app are consumed "over the Internet." If you're using Redis through an IaaS provider's offering (e.g. AWS ElastiCache, Google Cloud Memorystore) then you get the benefit of them being able to spawn an instance "outside" your project/VPC (i.e. having it be managed by them), but have it nevertheless routed to an IP address inside your VPC. That might be enough security for you, if you don't have any legal requirements saying otherwise. For some people, it's not, and they need TLS on top anyway. > cluster stuff Have you looked at how it's done? It's just ease-of-use tooling around the obvious thing to do to scale Redis: partitioning the keyspace onto distinct Redis instances, and then routing requests to partitions based on the key. It's not like Redis has suddenly become a multi-master consensus system like Zookeeper; the router logic isn't even in the server codebase!

rockostrich 6y ago

> do people really expose Redis on the internet?? You've never browsed unprotected IPs and ports huh? So many random redis instances just lying around.

benmmurphy 6y ago

Having TLS support in the main client is useful because AWS only supports AUTH if you enable TLS. Running Redis without AUTH can be kind of dangerous because Redis can kind of speak HTTP* (I think you can define custom commands to fix this) so if you have web hooks in your system and don't properly filter internal addresses then you might allow external parties to run Redis commands against your system. * it's been years since I looked at this so maybe Redis now ships with inbuilt protection against this.

all_blue_chucks 6y ago

> do people really expose Redis on the internet? Do people really still send database traffic unencrypted over unencrypted internal networks?

stouset 6y ago

> Now we have SSL (do people really expose Redis on the internet??). This is 2020. The "hard outer shell, soft chewy center" model of security is dead and it's not coming back. Modern datacenters and cloud deployments use mTLS (mutually-authenticated TLS) between every service, everywhere, all the time. There are some massive benefits to this. For starters, you can limit what services talk to one-another entirely through distribution and signing of keys. Yes, this adds a burden of complexity if you go that route. But suddenly you don't have to care as much about (for instance) many network-exploitable vulnerabilities in your services because someone with a foothold on your network can't even get talk to your service in the first place if they don't have the right TLS cert, which is only on the handful of machines and only readable by the specific services that are legitimately allowed to connect to it. This is a much stronger guarantee than firewalling alone (though you should also use firewalling), because multiple services can be running on a host but only the applications that are allowed to talk to your service will have read access to that key. On the flip side, you have stronger guarantees that the service you're connecting to really is the service you're expecting it to be. If you're storing sensitive information in Redis, you can know for sure that the port you've connected to is the right Redis and not another, less-sensitive application's.

notyourday 6y ago
I hear this in marketing presentation or company pitches all the time. After that I as a consultant get access to the network and apart from some test that a developer stood up nothing matches the glossy talk. Thanks god for Wireguard. It has truly been the savior deploying encrypted networks.
- jimmcslim 6y ago
  > After that I as a consultant get access to the network and apart from some test that a developer stood up nothing matches the glossy talk. Or in my case recently... someone has generated a root certificate for the internal CA that uses an insecure crypto scheme, and Chrome still throws up a security error requiring users to click past the warnings to access the site. "Can you generate and roll out a new cert please? This isn't really 'security'?" "Oh we will get to it, can you just use the one you already have?"

kawsper 6y ago

> do people really expose Redis on the internet?? Accidentally on a hobby-server, because docker-compose automatically opens up firewall ports.

Florin_Andrei 6y ago

Super-simple, good-enough things don't last too long. They die when the ecosystem changes, or when another super-simple player comes along and looks a bit more shiny. It's just how the world works. You have to conquer to survive.

vageli 6y ago

SSL/encrypted connections is a requirement in some regulatory frameworks. For example, transmitting PHI needs to be done over an encrypted connection.

apple4ever 6y ago

SSL/TLS is super important. I'm glad Redis added it. Now if only Varnish would get in the game.

manigandham 6y ago

If Redis is already good enough for you, what actually changes for you to start disliking it? You don't have to use any of those features.

redis_mlc 6y ago

> To my thinking, Redis fit very well in the "lightweight linux thing" category. It sounds like you don't follow Redis then. That ship sailed years ago. Redis has at least 10 major features in addition to the caching you're talking about, including search. Redis is a kind of database now. If you just want a cache, use memcached. Half of my jiras at one company were related to enabling SSL for Redis due to compliance reasons (all for internal use.) Now those can be closed.

dpipemazo 6y ago

We found streams to be a breakthrough feature for pub-sub type data on IoT devices. That it can both be low-latency pub-sub and a stateful, short-lived cache is quite powerful to improve performance for many queries to the types of data generated by cameras and high-frequency sensor devices. http://atomdocs.io/ http://atomdocs.io/ https://github.com/elementary-robotics/atom https://github.com/elementary-robotics/atom

__david__ 6y ago

Personally I'm jumping up and down for ACLs. I went so far as to implement a proof of concept Redis proxy that added ACLs a couple years ago, before I heard that they would be in 6. ACLs may be niche, but when you need 'em, you need 'em!

dagi3d 6y ago

The question would be if by adding those additional features, the experience to use the "basic/original" features got more cumbersome or the hardware requirements did change a lot. My guess it hasn't changed that much.

coleifer 6y ago

Replying to myself here. Of course every commenter wanted to bikeshed the SSL/crypto sentence. I forgot where I was commenting. I accidentally set the stage for a mountain of security circlejerking. No nibbles on the protocol point, which in fact was the main thing.