3 ms·

By saving bandwidth, do you mean saving network bandwidth between a Redis server and a client on another machine? In that case, another solution would be for th

By saving bandwidth, do you mean saving network bandwidth between a Redis server and a client on another machine? In that case, another solution would be for the user to write her own daemon (in a language of her choosing) that sits on the same machine as Redis, listens for "custom" commands from the remote client, and carries them out by communicating with the Redis server over a local socket. I will be interested to see how scripting Redis with Lua measures up to that solution. The separate daemon would have the overhead of protocol handling for each communication with Redis, but it could also be written in a compiled language, a language with better concurrency support, etc. It would also be trivially sandboxed from the Redis server.

antirez 15y ago
Teaspoon, this was definitely one of the ideas we had, but the performance hint of doing the I/O with another process is basically almost as bad as the socket in between. The problem is not just the amount of bandwidth (but it is one of the problems when there are a massive number of clients, we have an use case that I can't mention in detail about this). But also the problem is, the I/O is where most of the time is spent, and it is a shame. With Lua scripting we fix part of this problem allowing to deliver more performances. Proof is what it was possible to do with variable argument list push. You can push something like 350k items per second per core just using an LPUSH with a few more arguments per item.
- teaspoon 15y ago
  Makes sense to me. Do the Lua scripts get lower-level access to the data structures, too? E.g., could I write a variable-argument-list LINSERT that inserts M items in O(M+N) time rather than O(M*N)?
  - antirez 15y ago
    I don't think such low level access will be allowed. Would be cool but it is a lot of work and means to change the scripting layer every time we change the internals. I'll start with something much higher level than that.
jaksprats 15y ago
in the redis community we have also discussed having a daemon sitting next to the server and doing localhost redis communication. The downsides to doing this are it introduces an extra piece to break. W/ luajit2, which is almost as fast as post-JITted Java, the speed of the scripts are amazingly fast (Salvatore's current method of calling the scripts is dead on, it requires no interpretation, it maps directly to lua_p* calls). Disclaimer: I wrote AlchemyDB, embedded lua in redis a while back and have played w/ it constantly (it is robust and mind bogglingly flexible). As for the speed differences of embedded lua vs a daemon sitting next to the server, Alchemy has its test suite in Lua and some tests I run via an external client, and some tests I run internally via embedded Lua. The speed difference between client/daemon and embedded lua becomes VERY evident (10X faster) on large loops, where I/O and TCP kernel time are saved ... but as redis is single threaded server, scripts block the server during their execution, so it is just dangerous for novice programmers. All in all, if redis embeds lua correctly, it will really open up the project and it is a very minimal bloat, lua is tiny, and it is just one command :)

antirez 15y ago

Teaspoon, this was definitely one of the ideas we had, but the performance hint of doing the I/O with another process is basically almost as bad as the socket in between. The problem is not just the amount of bandwidth (but it is one of the problems when there are a massive number of clients, we have an use case that I can't mention in detail about this). But also the problem is, the I/O is where most of the time is spent, and it is a shame. With Lua scripting we fix part of this problem allowing to deliver more performances. Proof is what it was possible to do with variable argument list push. You can push something like 350k items per second per core just using an LPUSH with a few more arguments per item.

teaspoon 15y ago
Makes sense to me. Do the Lua scripts get lower-level access to the data structures, too? E.g., could I write a variable-argument-list LINSERT that inserts M items in O(M+N) time rather than O(M*N)?
- antirez 15y ago
  I don't think such low level access will be allowed. Would be cool but it is a lot of work and means to change the scripting layer every time we change the internals. I'll start with something much higher level than that.

jaksprats 15y ago

in the redis community we have also discussed having a daemon sitting next to the server and doing localhost redis communication. The downsides to doing this are it introduces an extra piece to break. W/ luajit2, which is almost as fast as post-JITted Java, the speed of the scripts are amazingly fast (Salvatore's current method of calling the scripts is dead on, it requires no interpretation, it maps directly to lua_p* calls). Disclaimer: I wrote AlchemyDB, embedded lua in redis a while back and have played w/ it constantly (it is robust and mind bogglingly flexible). As for the speed differences of embedded lua vs a daemon sitting next to the server, Alchemy has its test suite in Lua and some tests I run via an external client, and some tests I run internally via embedded Lua. The speed difference between client/daemon and embedded lua becomes VERY evident (10X faster) on large loops, where I/O and TCP kernel time are saved ... but as redis is single threaded server, scripts block the server during their execution, so it is just dangerous for novice programmers. All in all, if redis embeds lua correctly, it will really open up the project and it is a very minimal bloat, lua is tiny, and it is just one command :)