4 ms·

I'm trying to be cautious yet open-minded and optimistic with Redis modules. Redis is such a well-written C software that I can't help but think modules might

I'm trying to be cautious yet open-minded and optimistic with Redis modules. Redis is such a well-written C software that I can't help but think modules might become an antipattern to Redis. I like the idea that the option to use modules exists, but I hope it doesn't spiral into dependency hell as it is with the Ruby ecosystem. I hope library developers think very hard before employing and requiring a module. Redis is fast, easy to use, and extremely flexible, and makes it easy to use any of the hosted options (Redis Labs, RedisToGo). Its allure is that Redis is batteries-included yet small enough like a Swiss knife. I wouldn't mind installing a module if it brought in significant optimization for a specific use case. I personally hope most library developers will pretend modules don't exist, and that only app developers will use them. I have this selfish desire that most libraries stick to the vanilla Redis features, because for most things you can use app code to manipulate the features you need without modifying Redis and compromising performance. Modules might add to the heap of things to think about, such as special Redis deployments or the things that can go wrong with the module code. My fear is that instead of years of using the simple Lego blocks that is vanilla Redis, modules will start a pattern where we need to install special modules to use certain libraries. TL;DR, I'm worried for the potential for abuse with Redis modules. My love for Redis was how simple and powerful it is, and yet now Redis has grown up, ready for modules.

fzsix 10y ago
My hope is that the modules will be used more for common use cases (leaderboards, real-time counters, etc) instead of turning into gem dependency hell. Having a community backed module for a single purpose in that regard sounds pretty good, and the fact that they are written in C should filter out devs that follow a higher level dependency model as raw performance is a key metric.
- jasonjei 10y ago
  I agree. I think Sidekiq is a good example of how modules are used. You can add the Pro features, use a community plugin, but the plugins usually don't affect core functionality (such as the Sidekiq "cron" scheduler, error reporting options, etc). I think C is both a blessing and a curse. The challenge of using C correctly has led to memory leaks, remote code execution, segfaults, etc. I'd use Go if it's possible to write Redis modules.
  - PeCaN 10y ago
    You could use C++, Ada, or Rust if you use the right pragmas. I don't use Go but I'd be surprised if it can't export C ABI compatible functions somehow.
  - tyho 10y ago
    It is possible to make Go redis modules since go 1.5 thanks to the c-shared buildmode, I would be surprised if there wasn't a go package to make it easy within the month.
  - dvirsky 10y ago
    I was planning on writing a Go module wrapper but didn't get to it before RedisConf. I'm more interested though to see when a JS module wrapper gets implemented.
spinlock 10y ago
The only other C project I know of with a module system in Linux. I've actually found that Linux modules make development _much_ easier because you can isolate your code from the rest of the Linux code base. Of course, the last time I was professionally developing with the Linux kernel, git hadn't been invented yet and you really couldn't keep the entire kernel in version control without going insane. So, maybe things are different now.
- elcapitan 10y ago
  Are modules not pretty common? Apache and Nginx have modules like that as well.
  - deleted 10y ago
    [deleted]
  - spinlock 10y ago
    I was talking more of my experience than C projects in general. However, I'd expect a smaller percentage of C projects to support modules compared to Ruby given the differences in the language.
  - themartorana 10y ago
    Nginx's modularity and tolerance/embrace of plugins is absolutely amazing. I use a combination of openresty[0], lua scripting, and third party modules in our API servers (coincidentally, one that talks directly to Redis) and it's still one of the fastest, best pieces of software in our stack. I understand the concern, but if it ends up being half as flexible (and I already love me some Redis) I can only see good things.
- crdoconnor 10y ago
  There's varnish as well.
antirez 10y ago
I can totally sympathize with your feelings, but I believe that it depends a lot on how the Redis project will set the tone about this feature. Example of things that will make, in my opinion, the difference: 1. There will never be a way to install modules automatically provided by the Redis OSS project. The use case is to grab a few modules you may need for vertical use cases that are otherwise not cover, so you just don't install a number of modules depending on other modules and so forth. 2. I'll not write modules but will care about exporting a good modules interface. What I believe has to be done, will be implemented inside the Redis core, not outside. SO it's totally a community thing. For the project itself is like: "we are going to give you this possibility, so that the fact the core is conservative about what functions to add, does not block you". But the model for Redis does not change, Redis is what we provide in the core. I work only on core stuff. The site only documents what we have in the core. 3. I'm not going to debug for hours or days crashes happening with modules loaded. On crash the list of modules installed will be reported in the crash trace. So modules are going to be a good addition and there will be use cases that can be solved very well with modules, but are not going to change what Redis is. It's up to the user if to use Redis "core", or if to leverage modules. Wise users will pick modules when needed, and will inspect modules for quality and fit. Unwise users will install a lot of garbage, then eventually say Redis is a shit. But this is how the world works :-)
maerF0x0 10y ago
agree. Magento, wordpress and npm come to mind as places where quality is difficult to ascertain and can easily turn a great setup into a brick'd prod server.

fzsix 10y ago

My hope is that the modules will be used more for common use cases (leaderboards, real-time counters, etc) instead of turning into gem dependency hell. Having a community backed module for a single purpose in that regard sounds pretty good, and the fact that they are written in C should filter out devs that follow a higher level dependency model as raw performance is a key metric.

jasonjei 10y ago
I agree. I think Sidekiq is a good example of how modules are used. You can add the Pro features, use a community plugin, but the plugins usually don't affect core functionality (such as the Sidekiq "cron" scheduler, error reporting options, etc). I think C is both a blessing and a curse. The challenge of using C correctly has led to memory leaks, remote code execution, segfaults, etc. I'd use Go if it's possible to write Redis modules.
- PeCaN 10y ago
  You could use C++, Ada, or Rust if you use the right pragmas. I don't use Go but I'd be surprised if it can't export C ABI compatible functions somehow.
- tyho 10y ago
  It is possible to make Go redis modules since go 1.5 thanks to the c-shared buildmode, I would be surprised if there wasn't a go package to make it easy within the month.
- dvirsky 10y ago
  I was planning on writing a Go module wrapper but didn't get to it before RedisConf. I'm more interested though to see when a JS module wrapper gets implemented.

spinlock 10y ago

The only other C project I know of with a module system in Linux. I've actually found that Linux modules make development _much_ easier because you can isolate your code from the rest of the Linux code base. Of course, the last time I was professionally developing with the Linux kernel, git hadn't been invented yet and you really couldn't keep the entire kernel in version control without going insane. So, maybe things are different now.

elcapitan 10y ago
Are modules not pretty common? Apache and Nginx have modules like that as well.
- deleted 10y ago
  [deleted]
- spinlock 10y ago
  I was talking more of my experience than C projects in general. However, I'd expect a smaller percentage of C projects to support modules compared to Ruby given the differences in the language.
- themartorana 10y ago
  Nginx's modularity and tolerance/embrace of plugins is absolutely amazing. I use a combination of openresty[0], lua scripting, and third party modules in our API servers (coincidentally, one that talks directly to Redis) and it's still one of the fastest, best pieces of software in our stack. I understand the concern, but if it ends up being half as flexible (and I already love me some Redis) I can only see good things.
crdoconnor 10y ago
There's varnish as well.

antirez 10y ago

I can totally sympathize with your feelings, but I believe that it depends a lot on how the Redis project will set the tone about this feature. Example of things that will make, in my opinion, the difference: 1. There will never be a way to install modules automatically provided by the Redis OSS project. The use case is to grab a few modules you may need for vertical use cases that are otherwise not cover, so you just don't install a number of modules depending on other modules and so forth. 2. I'll not write modules but will care about exporting a good modules interface. What I believe has to be done, will be implemented inside the Redis core, not outside. SO it's totally a community thing. For the project itself is like: "we are going to give you this possibility, so that the fact the core is conservative about what functions to add, does not block you". But the model for Redis does not change, Redis is what we provide in the core. I work only on core stuff. The site only documents what we have in the core. 3. I'm not going to debug for hours or days crashes happening with modules loaded. On crash the list of modules installed will be reported in the crash trace. So modules are going to be a good addition and there will be use cases that can be solved very well with modules, but are not going to change what Redis is. It's up to the user if to use Redis "core", or if to leverage modules. Wise users will pick modules when needed, and will inspect modules for quality and fit. Unwise users will install a lot of garbage, then eventually say Redis is a shit. But this is how the world works :-)

maerF0x0 10y ago

agree. Magento, wordpress and npm come to mind as places where quality is difficult to ascertain and can easily turn a great setup into a brick'd prod server.