6 ms·

Lua: A Guide for Redis Users

123 points by ergot 10y ago

koolba 10y ago
This is the best line from the article: > The best scripts simply extend the existing Redis vocabulary of small atomic data operations with the smallest bit of logic necessary. Having used quite a bit of Lua (in Redis) I whole heartedly agree.
- john_reel 10y ago
  Yeah that’s generally how Lua should be used.
- manishsharan 10y ago
  I use redis + lua quite a bit in my home-grown analytics real time tool. My biggest fear is when do I know I am overusing lua ? Is there a quantitative measure for "smallest bit" of logic ? I fear I have been given a very long rope with Lua + Redis and I may be unknowingly about to hang myself.
  - ansible 10y ago
    I don't understand what your concern is. Lua is a relatively modern script language. It has all the tools you need to create modular programs in whatever style you like. Imperative is easy, OO style nearly so (there are plenty of OO implementations to choose from), and there is even some support for functional programming. The PUC-Rio (main) implementation has a fast bytecode interpreter, and LuaJIT is wicked fast compared to just about anything else. People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else.
    - vvanders 10y ago
      > People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else. We used it to write almost all the game-logic on a few titles. Ran great on all sorts of platforms(this was before LuaJIT). Coroutines are awesome for AI, lets you bundle in all sorts of implicit state easily and concisely.
  - marktangotango 10y ago
    > Is there a quantitative measure for "smallest bit" of logic ? I don't see that as a valid concern anymore. The main concerns are 1) size of script you can comprehend, debug, and maintain and 2) do the most with the fewest network/redis calls. If you're 200 line lua script replaces 5-10 individual redis calls, that's surely a win simply from the perspective of network latency and data locality.
joaomsa 10y ago
I keep a bunch of small scripts for common tasks like deleting or aggregating keys maching certain criteria. I wish redis offered a mechanism to save and call these scripts from the server, akin to the usual database functions and stored procedures. I wonder if this was a conscious decision or something that's in the pipeline eventually.
- ryanworl 10y ago
  What about this? http://redis.io/commands/script-load http://redis.io/commands/script-load http://redis.io/commands/evalsha http://redis.io/commands/evalsha EDIT: I realize this was mentioned in the article, just wondering why it doesn't fit your needs.
  - thezilch 10y ago
    I believe the ask is to have any scripts loaded to persist through restarts of Redis. As is, you have to reload them on each init of Redis. As such, the onus is on applications to catch evalsha errors and load the missing scripts, in the case of Redis being restarted.
    - unrealhoang 10y ago
      We built the same mechanism using HashMap and `SCRIPT LOAD` on restart/deployment to make a mapping table between function name and its SHA. Any client want to call the function just need to fetch the table and using the according SHA via `EVALSHA` command.
marktangotango 10y ago
This is a really good article, I referred to it a lot when working with Lua and Redis. Like a lot of people, I really admire antirez. The pragmatism he displayed choosing Lua was exceptional imo: >> I think that for what we need Lua beats everybody else hands down. The language is not one that I particularly like, compared to Ruby for instance, but who cares? We are programmers and can code a short script in any language we want, but the point is, Lua is a wonderful implementation. Easy to embed, without even a configure script, like Redis! And FAST.[1] [1] http://oldblog.antirez.com/post/redis-and-scripting.html http://oldblog.antirez.com/post/redis-and-scripting.html
- john_reel 10y ago
  Honestly, it’s a great article. That said, there’s nothing wrong with nils in tables in Lua if you know what you’re doing. Yes, they can introduce weird behaviors, (particularly with the # operator) but if you understand how next, pairs, ipairs, and # work, which are required if you want predictable table iteration behavior in Lua, then this shouldn’t be an issue. I think it’s dangerous to say that nils terminate tables if you’re not going to explain table iteration in Lua. Edit: Here's a picture example showing how easily nils can cause confusion if you're using a #table based loop. https://i.imgur.com/qW0XoSs.png https://i.imgur.com/qW0XoSs.png for i,v in pairs(table) will go through an entire table. for i,v in ipairs(table) will stop at a nil. It's meant for consistent behavior in tables with numerical indices.
  - scythe 10y ago
    The easiest way to have a value which "acts like null" but doesn't mess with table indexing is to use `false`, which is the only Lua value other than nil which is falsey (0 and "" are truthy).
    - amttc 10y ago
      Yeah, I wish Lua hadn't taken undefined and nil to be the same thing given the way tables work in it. Most of the time it doesn't matter, thankfully. Sometimes you bump into it. A sigil value can work too if you absolutely need it. ex: undef = {} function isundef(val) return undef == val end The sigil value is truthy and makes your code more convoluted, though. ex: if not val or isundef(val) then -- ... So, :/. I do love the language though.
idank 10y ago
Are redis scripts still recommended now that redis modules are a thing? One of the pain points of scripting is the lack of test coverage.
- pkulak 10y ago
  If I was just trying to group 3 operations together into a "transaction", a script would seem like the way to go. But yeah, as soon as my "script" got at all long/complicated, I'd take a long look at making it into a module.
  - mwpmaybe 10y ago
    Or you could use a Redis transaction for a transaction. :-) http://redis.io/topics/transactions http://redis.io/topics/transactions
- catwell 10y ago
  There is no reason why modules would be easier to test than scripts. It is probably the opposite. Modules are great when you want to use Redis as a server for something it does not support, but I would still use scripting for most things.
kawsper 10y ago
Is there a way to automatically load a script or a directory of scripts when the redis-server starts?
- justinhj 10y ago
  I don't think that's the recommended way of doing things. Redis will cache compiled Lua scripts internally so after the first EVAL you can call EVALSHA and pass the scripts hash. But you're client code should always be ready for the case that maybe the script is not there any more. In which case you call eval again. Automated tools that upload all the scripts would have no way to say "compile and cache this but don't run it" so you'd need your scripts to handle not doing anything. The usual pattern, then, is to simply send EVALSHA always and if it fails then call EVAL. With small numbers of scripts and larges numbers of operations this will be more efficient and doesn't require any co-ordination with the Redis server when scripts change or servers restart.
  - ryanworl 10y ago
    http://redis.io/commands/script-load http://redis.io/commands/script-load While I agree that the caller should be ready for a script not existing, one could build an automated tool to load scripts when starting redis using the SCRIPT LOAD command.
    - justinhj 10y ago
      Ah I missed that capability thanks for the correction
    - justinhj 10y ago
      Ah I missed that capability thanks for the correction
noahdesu 10y ago
I really like the way this article introduces the concepts of integrating Lua into a storage system like Redis. Shameless plug: the integration of Lua into Redis served as inspiration for our recent integration of Lua into Ceph/RADOS [1] for building custom object interfaces. I'll taking a cue from this article when we update our out-of-date documentation on the feature [2]. [1]: https://github.com/ceph/ceph/pull/7338 https://github.com/ceph/ceph/pull/7338 [2]: http://noahdesu.github.io/2014/01/22/dynamic-rados-object-interfaces.html http://noahdesu.github.io/2014/01/22/dynamic-rados-object-in...
keyle 10y ago
Tangeant: it's so obnoxious that every code snippet from Github has to have such a signature. All I could read was "hosted with <3 by GitHub"
- deleted 10y ago
  [deleted]
- scrollaway 10y ago
  That's probably because it's not what gist is made for. Embedding a gist instead of adding a fenced block in your markdown or whatever blog engine you're using is kinda silly for a couple of lines of code. https://gist.github.com/bpo/044fed4c53fc61a1a0a0 https://gist.github.com/bpo/044fed4c53fc61a1a0a0
- _bpo 10y ago
  Author here. I completely agree. Embedded gists didn't have this annoyance when I wrote the article. I figured GitHub could be relied on to not change things up too much, but that's what I get for depending on an external service...

123 points by ergot 10y ago

This is the best line from the article: > The best scripts simply extend the existing Redis vocabulary of small atomic data operations with the smallest bit of logic necessary. Having used quite a bit of Lua (in Redis) I whole heartedly agree.

john_reel 10y ago
Yeah that’s generally how Lua should be used.
manishsharan 10y ago
I use redis + lua quite a bit in my home-grown analytics real time tool. My biggest fear is when do I know I am overusing lua ? Is there a quantitative measure for "smallest bit" of logic ? I fear I have been given a very long rope with Lua + Redis and I may be unknowingly about to hang myself.
- ansible 10y ago
  I don't understand what your concern is. Lua is a relatively modern script language. It has all the tools you need to create modular programs in whatever style you like. Imperative is easy, OO style nearly so (there are plenty of OO implementations to choose from), and there is even some support for functional programming. The PUC-Rio (main) implementation has a fast bytecode interpreter, and LuaJIT is wicked fast compared to just about anything else. People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else.
  - vvanders 10y ago
    > People don't tend to write really huge applications in Lua, but that is more due to the lack of libraries rather than anything else. We used it to write almost all the game-logic on a few titles. Ran great on all sorts of platforms(this was before LuaJIT). Coroutines are awesome for AI, lets you bundle in all sorts of implicit state easily and concisely.
- marktangotango 10y ago
  > Is there a quantitative measure for "smallest bit" of logic ? I don't see that as a valid concern anymore. The main concerns are 1) size of script you can comprehend, debug, and maintain and 2) do the most with the fewest network/redis calls. If you're 200 line lua script replaces 5-10 individual redis calls, that's surely a win simply from the perspective of network latency and data locality.

joaomsa 10y ago

I keep a bunch of small scripts for common tasks like deleting or aggregating keys maching certain criteria. I wish redis offered a mechanism to save and call these scripts from the server, akin to the usual database functions and stored procedures. I wonder if this was a conscious decision or something that's in the pipeline eventually.

ryanworl 10y ago
What about this? http://redis.io/commands/script-load http://redis.io/commands/script-load http://redis.io/commands/evalsha http://redis.io/commands/evalsha EDIT: I realize this was mentioned in the article, just wondering why it doesn't fit your needs.
- thezilch 10y ago
  I believe the ask is to have any scripts loaded to persist through restarts of Redis. As is, you have to reload them on each init of Redis. As such, the onus is on applications to catch evalsha errors and load the missing scripts, in the case of Redis being restarted.
  - unrealhoang 10y ago
    We built the same mechanism using HashMap and `SCRIPT LOAD` on restart/deployment to make a mapping table between function name and its SHA. Any client want to call the function just need to fetch the table and using the according SHA via `EVALSHA` command.

marktangotango 10y ago

This is a really good article, I referred to it a lot when working with Lua and Redis. Like a lot of people, I really admire antirez. The pragmatism he displayed choosing Lua was exceptional imo: >> I think that for what we need Lua beats everybody else hands down. The language is not one that I particularly like, compared to Ruby for instance, but who cares? We are programmers and can code a short script in any language we want, but the point is, Lua is a wonderful implementation. Easy to embed, without even a configure script, like Redis! And FAST.[1] [1] http://oldblog.antirez.com/post/redis-and-scripting.html http://oldblog.antirez.com/post/redis-and-scripting.html

john_reel 10y ago
Honestly, it’s a great article. That said, there’s nothing wrong with nils in tables in Lua if you know what you’re doing. Yes, they can introduce weird behaviors, (particularly with the # operator) but if you understand how next, pairs, ipairs, and # work, which are required if you want predictable table iteration behavior in Lua, then this shouldn’t be an issue. I think it’s dangerous to say that nils terminate tables if you’re not going to explain table iteration in Lua. Edit: Here's a picture example showing how easily nils can cause confusion if you're using a #table based loop. https://i.imgur.com/qW0XoSs.png https://i.imgur.com/qW0XoSs.png for i,v in pairs(table) will go through an entire table. for i,v in ipairs(table) will stop at a nil. It's meant for consistent behavior in tables with numerical indices.
- scythe 10y ago
  The easiest way to have a value which "acts like null" but doesn't mess with table indexing is to use `false`, which is the only Lua value other than nil which is falsey (0 and "" are truthy).
  - amttc 10y ago
    Yeah, I wish Lua hadn't taken undefined and nil to be the same thing given the way tables work in it. Most of the time it doesn't matter, thankfully. Sometimes you bump into it. A sigil value can work too if you absolutely need it. ex: undef = {} function isundef(val) return undef == val end The sigil value is truthy and makes your code more convoluted, though. ex: if not val or isundef(val) then -- ... So, :/. I do love the language though.

idank 10y ago

Are redis scripts still recommended now that redis modules are a thing? One of the pain points of scripting is the lack of test coverage.

pkulak 10y ago
If I was just trying to group 3 operations together into a "transaction", a script would seem like the way to go. But yeah, as soon as my "script" got at all long/complicated, I'd take a long look at making it into a module.
- mwpmaybe 10y ago
  Or you could use a Redis transaction for a transaction. :-) http://redis.io/topics/transactions http://redis.io/topics/transactions
catwell 10y ago
There is no reason why modules would be easier to test than scripts. It is probably the opposite. Modules are great when you want to use Redis as a server for something it does not support, but I would still use scripting for most things.

kawsper 10y ago

Is there a way to automatically load a script or a directory of scripts when the redis-server starts?

justinhj 10y ago
I don't think that's the recommended way of doing things. Redis will cache compiled Lua scripts internally so after the first EVAL you can call EVALSHA and pass the scripts hash. But you're client code should always be ready for the case that maybe the script is not there any more. In which case you call eval again. Automated tools that upload all the scripts would have no way to say "compile and cache this but don't run it" so you'd need your scripts to handle not doing anything. The usual pattern, then, is to simply send EVALSHA always and if it fails then call EVAL. With small numbers of scripts and larges numbers of operations this will be more efficient and doesn't require any co-ordination with the Redis server when scripts change or servers restart.
- ryanworl 10y ago
  http://redis.io/commands/script-load http://redis.io/commands/script-load While I agree that the caller should be ready for a script not existing, one could build an automated tool to load scripts when starting redis using the SCRIPT LOAD command.
  - justinhj 10y ago
    Ah I missed that capability thanks for the correction
  - justinhj 10y ago
    Ah I missed that capability thanks for the correction

noahdesu 10y ago

I really like the way this article introduces the concepts of integrating Lua into a storage system like Redis. Shameless plug: the integration of Lua into Redis served as inspiration for our recent integration of Lua into Ceph/RADOS [1] for building custom object interfaces. I'll taking a cue from this article when we update our out-of-date documentation on the feature [2]. [1]: https://github.com/ceph/ceph/pull/7338 https://github.com/ceph/ceph/pull/7338 [2]: http://noahdesu.github.io/2014/01/22/dynamic-rados-object-interfaces.html http://noahdesu.github.io/2014/01/22/dynamic-rados-object-in...

keyle 10y ago

Tangeant: it's so obnoxious that every code snippet from Github has to have such a signature. All I could read was "hosted with <3 by GitHub"

deleted 10y ago
[deleted]
scrollaway 10y ago
That's probably because it's not what gist is made for. Embedding a gist instead of adding a fenced block in your markdown or whatever blog engine you're using is kinda silly for a couple of lines of code. https://gist.github.com/bpo/044fed4c53fc61a1a0a0 https://gist.github.com/bpo/044fed4c53fc61a1a0a0
_bpo 10y ago
Author here. I completely agree. Embedded gists didn't have this annoyance when I wrote the article. I figured GitHub could be relied on to not change things up too much, but that's what I get for depending on an external service...