5 ms·
An Afternoon of Code Golf in Lua to Achieve 4x Performance in Redis
- kortox 10y agoHi, I'm the author of the post. Happy to answer any questions or criticisms folks have.
- notliketherest 10y agoWhat is Code Golf?
- detaro 10y agoFrom the link in the article: "Code golf is a type of recreational computer programming competition in which participants strive to achieve the shortest possible source code that implements a certain algorithm." https://en.wikipedia.org/wiki/Code_golf https://en.wikipedia.org/wiki/Code_golf There is an entire StackExchange site for it, which is interesting to look around in: https://codegolf.stackexchange.com https://codegolf.stackexchange.com
- rakoo 10y agoHi, that's a very interesting post, thanks a lot ! It's cool to see that sometimes a small lua script can do exactly what you need, instead of adding a new Redis command to the already very long list. I have a question regarding "deployment" of Lua scripts (I've never used Redis for more than toys, so I may be wrong on something). I would assume a caller would do something like this just after the Redis cluster is up: - calculate script's SHA - EVALSHA a script - script is not there - EVAL script And then later in the lifetime of the caller: - calculate script's SHA - EVALSHA script Which means you wouldn't need to care about the script size, nor would you need to fiddle with your deployment pipeline. Am I missing something ?
- kortox 10y agoHi, thats a good point and the approach you describe would work for a single Redis instance. In our case, we were using Twemproxy in front of Redis and it doesn't support Redis's "SCRIPT LOAD" command so we would need a way to load the script on each Redis node ahead of time. One other concern when using EVAL and EVALSHA through Twemproxy is guaranteeing that all keys in the command map to the same Redis node, in our case we only had one key so wasn't a concern. Documentation on that can be found here https://github.com/twitter/twemproxy/blob/master/notes/redis.md#scripting https://github.com/twitter/twemproxy/blob/master/notes/redis...
- evv555 10y agoWhat did you you use to make the flowcharts?
- kortox 10y agosimplediagrams
- paladin314159 10y agoI really wonder why redis hasn't implemented HMINCRBY. Have run into that issue a bunch of times and the Github issue just doesn't convince me.
- antirez 10y agoIt's a commonly used primitive and I'm tempted to do it sometimes, but the problem is that there is no clear semantics. For instance in the blog post, apparently, the author is interested in incrementing multiple fields of the same key, while the proposal on the Github issue, wanted to increment different keys. Unfortunately to extend HINCRBY to make it variadic is hard because the return value would change... At the same time some users may absolutely not want to have all the items returned after the increment, since if you increment a batch of 1000 items and are not interested in the result, to get a 1000 items reply is not great. Given all this problems, and in order to avoid introducing new commands, probably what could be done is to introduce options into HICNRBY, so that it can model both the variadic thing (only switching to a different reply when some option is given) and also to have specific options in order to switch on/off the reply of the incremented values and so forth. Commands designed recently, after the old errors, have such capabilities. For instance BITFIELD is pretty powerful, and is a command doing increments: https://redis.io/commands/bitfield https://redis.io/commands/bitfield
- paladin314159 10y agoThanks for responding (and for Redis)! That makes a lot more sense now, would love to see the older commands get new, more powerful versions :)
- partycoder 10y agoRunning MONITOR in production can severely affect performance, by 50% per monitoring client according to the documentation. twemproxy independently exposes some statistics, it's probably safer to do it that way.
- kortox 10y agoHi, I'm the author of the post. I'll have to look into getting the metrics from Twemproxy in the future. I typically only run MONITOR for a few seconds 'cause I'm usually just looking for high volume commands & keys.
- antirez 10y agoOP may want to try "INFO commandstats" to get the same metrics in a much cheaper way. Example output: # Commandstats cmdstat_info:calls=1,usec=57,usec_per_call=57.00 cmdstat_cluster:calls=8,usec=211,usec_per_call=26.38 cmdstat_zrange:calls=4,usec=1990,usec_per_call=497.50
- partycoder 10y agoExcellent remark. Note that antirez is Salvatore Sanfilippo, the author of redis. Kudos
- tdj 10y agoOne common pattern I had to frequently use was HSET of the same key on multiple hashmaps and different values. Not quite HMSET, but MHSET was a good name for it.