4 ms·

What is the use-case for server side scripting? Does a running script block readers and writers?

SpikeGronim 15y ago
Wavii is using LUA in Redis to implement extremely simple single object atomic consistency. Some but not all of the Redis data structure types support "do XYZ only if the key exists". For example lists support LPUSHX but there is no "X" equivalent for sorted sets. With LUA I can add the ZADDX command in about 4 lines of LUA and 4 lines of Ruby in my Redis driver. We have lots of sorted sets backed by the database. Suppose we add an item to the set. If the set is in Redis we want to do the O(logN) ZADD, if it is not in Redis we need to do an O(N) reload from the DB. The use of ZADDX + transactions in our DB lets multiple writers contend on adding an item to a sorted set and guarantee that at the end all the items are in the sorted set while doing O(logN) operations almost all the time.
antirez 15y ago
Every command blocks every other command in Redis, because Redis is single threaded. However the implementation of scripting in Redis does a good use of Lua, so a command implemented in Lua does not run too much slower than a command implemented in C, for simple tasks (since we use a single Lua interpreter, and cache compiled functions). So this is an example of what you can do with Lua scripting: redis 127.0.0.1:6379> set foo '{"b":20,"a":10}' OK The user wants to increment the value of "b" with a single query to Redis in an atomic way: $ cat /tmp/script.lua obj = cjson.decode(redis.call("GET",KEYS[1])) obj.b = obj.b + 1 redis.call("SET",KEYS[1],cjson.encode(obj)) return obj.b $ redis-cli --eval /tmp/script.lua foo (integer) 21 $ redis-cli --eval /tmp/script.lua foo (integer) 22 $ redis-cli GET foo "{\"a\":10,\"b\":22}" No back and forth from client to server, no locking needed. Basically you can implement your specialized, atomic commands, in Lua.
- kijin 15y ago
  The example you cited just looks like an awfully convoluted way to recreate HINCRBY. Obviously there are much more powerful things that you could do with scripting, but this example makes me wonder if all those H-commands might be replaced with JSON & scripting (or handled internally by JSON & scripting) some time in the future.
  - antirez 15y ago
    Well it's not really HINCRBY, it increments a field of a JSON object that is otherwise not supported by Redis natively. It was just an example, but it is not a strange one, because there are many users with JSON objects, and similarly there are many users with specific needs: Do this if there is that in the key, add this element only if the list is < N elements, and so forth. With Lua scripting you can model all this without back-and-forth without the client and server, in other words, with 1/100 of the latency (or less). Given that we'll implement new commands only when they are "fundamental" operatons. For instance there is some plan to implement list splicing. When this happens it's worth to write it in C for max performances, so I think we'll unlikely write parts of Redis in Lua itself. Scripting systems is something I love (see Emacs for instance) but IMHO Redis is the instance of system where this does not work: space and time are too important for Redis.
    - kijin 15y ago
      Thanks for the clarification. It's great to hear that scripting will reduce the need to create new commands. I'm sure some people have been getting slightly annoyed with the rapid proliferation of Redis commands, which already number well over 100. Can't remember them all!
      - xnxn 15y ago
        That reminds me, I'd really like to see an official wiki (or just a simple pastebin-like thing like the Flask snippets site) for Redis scripts, use cases, best practices... I've seen quite a few blog posts that highlight some amazing way to use Redis and it would be very helpful to have them all in one place.
        edanm 15y ago
        I agree. But I would also like to add that Redis's documentation is one of the best I've seen anywhere. Incredibly easy to find information about any command.
        catwell 15y ago

Wavii is using LUA in Redis to implement extremely simple single object atomic consistency. Some but not all of the Redis data structure types support "do XYZ only if the key exists". For example lists support LPUSHX but there is no "X" equivalent for sorted sets. With LUA I can add the ZADDX command in about 4 lines of LUA and 4 lines of Ruby in my Redis driver. We have lots of sorted sets backed by the database. Suppose we add an item to the set. If the set is in Redis we want to do the O(logN) ZADD, if it is not in Redis we need to do an O(N) reload from the DB. The use of ZADDX + transactions in our DB lets multiple writers contend on adding an item to a sorted set and guarantee that at the end all the items are in the sorted set while doing O(logN) operations almost all the time.

antirez 15y ago

Every command blocks every other command in Redis, because Redis is single threaded. However the implementation of scripting in Redis does a good use of Lua, so a command implemented in Lua does not run too much slower than a command implemented in C, for simple tasks (since we use a single Lua interpreter, and cache compiled functions). So this is an example of what you can do with Lua scripting: redis 127.0.0.1:6379> set foo '{"b":20,"a":10}' OK The user wants to increment the value of "b" with a single query to Redis in an atomic way: $ cat /tmp/script.lua obj = cjson.decode(redis.call("GET",KEYS[1])) obj.b = obj.b + 1 redis.call("SET",KEYS[1],cjson.encode(obj)) return obj.b $ redis-cli --eval /tmp/script.lua foo (integer) 21 $ redis-cli --eval /tmp/script.lua foo (integer) 22 $ redis-cli GET foo "{\"a\":10,\"b\":22}" No back and forth from client to server, no locking needed. Basically you can implement your specialized, atomic commands, in Lua.

kijin 15y ago
The example you cited just looks like an awfully convoluted way to recreate HINCRBY. Obviously there are much more powerful things that you could do with scripting, but this example makes me wonder if all those H-commands might be replaced with JSON & scripting (or handled internally by JSON & scripting) some time in the future.
- antirez 15y ago
  Well it's not really HINCRBY, it increments a field of a JSON object that is otherwise not supported by Redis natively. It was just an example, but it is not a strange one, because there are many users with JSON objects, and similarly there are many users with specific needs: Do this if there is that in the key, add this element only if the list is < N elements, and so forth. With Lua scripting you can model all this without back-and-forth without the client and server, in other words, with 1/100 of the latency (or less). Given that we'll implement new commands only when they are "fundamental" operatons. For instance there is some plan to implement list splicing. When this happens it's worth to write it in C for max performances, so I think we'll unlikely write parts of Redis in Lua itself. Scripting systems is something I love (see Emacs for instance) but IMHO Redis is the instance of system where this does not work: space and time are too important for Redis.
  - kijin 15y ago
    Thanks for the clarification. It's great to hear that scripting will reduce the need to create new commands. I'm sure some people have been getting slightly annoyed with the rapid proliferation of Redis commands, which already number well over 100. Can't remember them all!
    - xnxn 15y ago
      That reminds me, I'd really like to see an official wiki (or just a simple pastebin-like thing like the Flask snippets site) for Redis scripts, use cases, best practices... I've seen quite a few blog posts that highlight some amazing way to use Redis and it would be very helpful to have them all in one place.
      - edanm 15y ago
        I agree. But I would also like to add that Redis's documentation is one of the best I've seen anywhere. Incredibly easy to find information about any command.
      - catwell 15y ago