3 ms·

There seems to be a chronic need in the Redis community to avoid confrontation (or even be bucketed) in the same space as Kafka. Redis Streams is probably one

There seems to be a chronic need in the Redis community to avoid confrontation (or even be bucketed) in the same space as Kafka. Redis Streams is probably one of the most path-breaking things that Redis is doing - and can completely change what it stands for. That's not a bad thing..but it doesn't even get a title by itself but is rather called "data structures". Here's a question - do you anticipate competing with Kafka? There are a lot of us that are cheering Redis on for this...but it seems there is internal reluctance. It's like the long standing "Redis is not memcached" statement - even WordPress has a "choose Redis or memcached for object caching" option.

derefr 8y ago
> It's like the long standing "Redis is not memcached" statement - even WordPress has a "choose Redis or memcached for object caching" option. As was said in the post, Redis isn't memcached, but "a bunch of Redis nodes on the same machine together with a Redis Cluster proxy" is kind of equivalent to memcached. Redis and memcached are on different architectural levels—one Redis instance represents one shard of a sharded data-store with an IO-concurrency of r=1 w=1, while one memcached instance represents a complete data-store with an IO-concurrency of r=N w=1. To build a caching "virtual appliance" for deploying on some big hardware, a memcached appliance would just be a memcached instance; while a Redis appliance would contain N Redis nodes + 1 Redis Cluster proxy. It's a bit like comparing, say, UDP with TCP. There are features you can layer on top of UDP to make it act equivalently to TCP (as e.g. QUIC does), but you can also just use UDP by itself. UDP is not designed to be TCP or "compete" with TCP; but UDP happens to be able to be used—when you add extra stuff on top—for the same things TCP can be used for. Redis, like UDP, is a flexible, low-level "infrastructure component" that can be customized into tools to solve any number of use-cases—but, like UDP, Redis sets out to address only a very limited set of use-cases when used as a standalone "tool" without such customization. I personally think of Redis a lot like an OS distribution—you could make any number of virtual appliances by customizing e.g. Debian with your own extra packages and building the resulting appliance-nodes into your desired architecture; and you can make any number of data-store servers (solving any number of problems) by customizing Redis with your own commands, and then connecting the resulting nodes into your desired architecture. Redis itself isn't going to compete directly with Kafka. Kafka, like memcached, is on a higher architectural layer than Redis. But there's nothing stopping some downstream developer from building something on that layer, e.g. a "RedisMQ" server customized into a Kafka-killer.
- sandGorgon 8y ago
  >Redis itself isn't going to compete directly with Kafka. But there's nothing stopping some downstream developer from building something So this is a philosophical rather than a technical thing. For a hobbyist open source project, I would agree with you there - there's only so much resources that someone can carve out of his day job. For a company that raised a 60mil venture funding and specifically created a license to prevent IP leakage, this is disappointing. I want to pay Redis and I don't mind their licensing. But I'm then questioning the philosophy if everything meaningful has to be done by unpaid open-source developers/startups, who might be on tenuous footing to do anything meaningful with it because of the licensing.
  - derefr 8y ago
    I think you're confusing Redis with Redis Labs. Redis is built by antirez as a "hobbyist open source project." It is a small core, and will stay that way. It is an infrastructure component. It is fully open-source. Redis Labs is the company that raised 60mil venture funding, and Redis Labs' business model is exactly to create such "distributions" of Redis for their customers. These distributions are not open-source; they are the thing that has the weird license. Redis Labs does not own or control Redis. antirez owns and controls Redis. You could create a similar company of your own which is also selling Redis distributions (as a product, as a service, etc.) and make money off of doing so. Since Redis itself—the thing antirez develops—is open-source, nothing stops you from doing this, just like nothing stops Redis Labs from doing this. No "unpaid open-source developers" are needed. antirez just happens to work for Redis Labs—but not in the sense that they can really tell him what to do. They just want to pay him to ensure he keeps making it, because their business depends on the continued health of the open-source project at its core. (It's about the same situation as Guido van Rossum working for Dropbox, or Yukihiro Matsumoto working for Heroku.) If you want to talk about the well-being of some project, downstream of Redis, to build a full-featured multipurpose DBMS "tool" with Redis at its core, that is fully open-source... well, that project doesn't exist. Redis Labs is not(!) that project; Redis Labs is a services company. But there's nothing stopping such a project from existing, and there's nothing stopping a foundation springing up around it to take care of it and its developers (like e.g. the Linux Software Foundation, or the PostgreSQL EU+US foundations.) But that software project, if it existed, would no more be "the Redis project" than https://openresty.org/en/ https://openresty.org/en/ is "the Nginx project." It'd just be a customized distribution, by separate people.
antirez 8y ago
It's true, I don't like to compare Redis to other products. I just want users to understand when Redis is a good candidate for their project and evaluate it. There is something that's too much of a taboo for me to compare what I did with what other did, I like to think more like if there are all this software that try to do things without trying to win one over the other, and are the developers out there to vote with their usage.
- sandGorgon 8y ago
  Sure thing. I respect that. But what is happening is that is causing a wide reluctance to even play in the same space. We have been super excited about Redis Streams for quite some time now. But we are not seeing any implementation usecases, tools...or even your awesome blog posts around streaming usecases. It would be great to see some more literature on the streaming stuff and things that you can (and are ) doing with it...and giving it a top level importance (beyond just a new data structure). This is super awesome stuff !

derefr 8y ago

> It's like the long standing "Redis is not memcached" statement - even WordPress has a "choose Redis or memcached for object caching" option. As was said in the post, Redis isn't memcached, but "a bunch of Redis nodes on the same machine together with a Redis Cluster proxy" is kind of equivalent to memcached. Redis and memcached are on different architectural levels—one Redis instance represents one shard of a sharded data-store with an IO-concurrency of r=1 w=1, while one memcached instance represents a complete data-store with an IO-concurrency of r=N w=1. To build a caching "virtual appliance" for deploying on some big hardware, a memcached appliance would just be a memcached instance; while a Redis appliance would contain N Redis nodes + 1 Redis Cluster proxy. It's a bit like comparing, say, UDP with TCP. There are features you can layer on top of UDP to make it act equivalently to TCP (as e.g. QUIC does), but you can also just use UDP by itself. UDP is not designed to be TCP or "compete" with TCP; but UDP happens to be able to be used—when you add extra stuff on top—for the same things TCP can be used for. Redis, like UDP, is a flexible, low-level "infrastructure component" that can be customized into tools to solve any number of use-cases—but, like UDP, Redis sets out to address only a very limited set of use-cases when used as a standalone "tool" without such customization. I personally think of Redis a lot like an OS distribution—you could make any number of virtual appliances by customizing e.g. Debian with your own extra packages and building the resulting appliance-nodes into your desired architecture; and you can make any number of data-store servers (solving any number of problems) by customizing Redis with your own commands, and then connecting the resulting nodes into your desired architecture. Redis itself isn't going to compete directly with Kafka. Kafka, like memcached, is on a higher architectural layer than Redis. But there's nothing stopping some downstream developer from building something on that layer, e.g. a "RedisMQ" server customized into a Kafka-killer.

sandGorgon 8y ago
>Redis itself isn't going to compete directly with Kafka. But there's nothing stopping some downstream developer from building something So this is a philosophical rather than a technical thing. For a hobbyist open source project, I would agree with you there - there's only so much resources that someone can carve out of his day job. For a company that raised a 60mil venture funding and specifically created a license to prevent IP leakage, this is disappointing. I want to pay Redis and I don't mind their licensing. But I'm then questioning the philosophy if everything meaningful has to be done by unpaid open-source developers/startups, who might be on tenuous footing to do anything meaningful with it because of the licensing.
- derefr 8y ago
  I think you're confusing Redis with Redis Labs. Redis is built by antirez as a "hobbyist open source project." It is a small core, and will stay that way. It is an infrastructure component. It is fully open-source. Redis Labs is the company that raised 60mil venture funding, and Redis Labs' business model is exactly to create such "distributions" of Redis for their customers. These distributions are not open-source; they are the thing that has the weird license. Redis Labs does not own or control Redis. antirez owns and controls Redis. You could create a similar company of your own which is also selling Redis distributions (as a product, as a service, etc.) and make money off of doing so. Since Redis itself—the thing antirez develops—is open-source, nothing stops you from doing this, just like nothing stops Redis Labs from doing this. No "unpaid open-source developers" are needed. antirez just happens to work for Redis Labs—but not in the sense that they can really tell him what to do. They just want to pay him to ensure he keeps making it, because their business depends on the continued health of the open-source project at its core. (It's about the same situation as Guido van Rossum working for Dropbox, or Yukihiro Matsumoto working for Heroku.) If you want to talk about the well-being of some project, downstream of Redis, to build a full-featured multipurpose DBMS "tool" with Redis at its core, that is fully open-source... well, that project doesn't exist. Redis Labs is not(!) that project; Redis Labs is a services company. But there's nothing stopping such a project from existing, and there's nothing stopping a foundation springing up around it to take care of it and its developers (like e.g. the Linux Software Foundation, or the PostgreSQL EU+US foundations.) But that software project, if it existed, would no more be "the Redis project" than https://openresty.org/en/ https://openresty.org/en/ is "the Nginx project." It'd just be a customized distribution, by separate people.

antirez 8y ago

It's true, I don't like to compare Redis to other products. I just want users to understand when Redis is a good candidate for their project and evaluate it. There is something that's too much of a taboo for me to compare what I did with what other did, I like to think more like if there are all this software that try to do things without trying to win one over the other, and are the developers out there to vote with their usage.

sandGorgon 8y ago
Sure thing. I respect that. But what is happening is that is causing a wide reluctance to even play in the same space. We have been super excited about Redis Streams for quite some time now. But we are not seeing any implementation usecases, tools...or even your awesome blog posts around streaming usecases. It would be great to see some more literature on the streaming stuff and things that you can (and are ) doing with it...and giving it a top level importance (beyond just a new data structure). This is super awesome stuff !