5 ms·

The Great Redis Misapprehension

65 points by dickeytk 15y ago

tabbyjabby 15y ago
Hmm. The author's recommendation to use Redis' 'KEYS' command to pattern match keys for invalidation is a dangerous one. 'KEYS' runs in O(n) time. If you're using Redis in a serious production environment, you do not want want to be using an O(n) command each time you need to invalidate a group of keys. It's better to group related items in a Redis hash table, and that way they are all stored under one common key. Honestly, I think this article is a more impoverished version of antirez's post on the same topic [1]. antirez, being one of the principal authors of Redis, is a much more authoritative source, and he actually describes all the patterns that this author described in greater detail. [1] http://antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html http://antirez.com/post/take-advantage-of-redis-adding-it-to...
- LeafStorm 15y ago
  And the O(N) is against your entire keyset, and not just the keys you're erasing. Which makes it even worse.
- dickeytk 15y ago
  Thanks for linking the antirez article, I actually haven't seen that one before I wrote the article. I put a link to it in the article. As far as they keys thing, it's obviously been a major point of contention with my article. I know the warning, I just don't think it's actually a problem for cache invalidation. See below on another comment of someone having similar concerns for my explanation why.
- dickeytk 15y ago
  tabby, I've also updated the article to mention the fact that KEYS may not be a solid use of the system. I would appreciate any feedback!
  - tabbyjabby 15y ago
    Sorry, I didn't mean to come across as a dick. I'm glad that you've taken the feedback of your readers into consideration. I'm a big fan of Redis, and it's a key component of our stack. Sorted sets are useful for a lot more than just leader boards, though that is a good use case for them. It's a bit late here, so I'm not feeling up to writing a big post, but I'm considering writing my own blog post on my experiences with Redis.
    - dickeytk 15y ago
      I don't think you were at all. Many people mentioned the same thing you did. I'm new at blogging, so I'm trying to take in everything and produce the best content I can, so I love the feedback! I would love to see a follow-up article too by the way!
philjackson 15y ago
"It has some persistence support, but does not appear to be super durable. If you're thinking of using like that though, you're misunderstanding the tool." This warrants a much more detailed explanation. The author should have spoken about possible durability options, like the append-only file, which, given the right configuration, makes Redis "fully-durable" at the inevitable sacrifice of some speed. A more informative read would be: http://redis.io/topics/persistence http://redis.io/topics/persistence
typicalrunt 15y ago
Now, this is where Redis comes in. You can find keys on wildcards! So you can just query it like so: keys post/83/* No no no. This is slow and there is a large warning section in the notes (http://redis.io/commands/keys http://redis.io/commands/keys) about using this in production environments. [Edit: Why is this bad? Think about if you have millions of keys in your environment. KEYS will need to iterate over a million keys to find the ones that match your pattern.] As an alternative to using KEYS, Redis provides the hash object. Store everything abou an object in a hash (e.g.: "posts:83") and then just delete the hash object. Everything under it will be removed as well. If you need to know what's in the hash that is going to be deleted, use the HKEYS command (which has no such warning about performance).
- dickeytk 15y ago
  Right, I know what the docs say, however, it's fine to use this for cache invalidation (but maybe not other use-cases) in production. The docs also say: "While the time complexity for this operation is O(N), the constant times are fairly low. For example, Redis running on an entry level laptop can scan a 1 million key database in 40 milliseconds." If you have so many keys that this is an issue for cache invalidation, you should be using memcached anyways. (Since it can be distributed, where in Redis distribution is left up to you to figure out) I wasn't able to dig it up, but I know I read an article about some consulting group making a site for a major shoe company where they did exactly this. Your hash method isn't the best way either though since it's more efficient to store everything in individual key-value pairs, and hashes cannot be nested. Really the BEST way to do this in Redis is to use a set containing all they keys related to an object, then clear each of them out when destroying an item.
  - dickeytk 15y ago
    It's no authoritative source, but I'm obviously not the only one that thinks this: http://www.webdevelopment-blog.com/2011/07/puma-on-redis/ http://www.webdevelopment-blog.com/2011/07/puma-on-redis/
    - tabbyjabby 15y ago
Mikushi 15y ago
Valid article, mostly. But the point is still valid, and i feel that most people using Redis do not get it, it is a data structure server, it's not that hard to understand... Use it as such and you won't be disappointed.
- dickeytk 15y ago
  Yes!! Thank you so much. That's exactly the point I was trying to make.
pkulak 15y ago
If I was greenfielding a new project tomorrow, I'd use a Redis/MySQL combo. MySQL for all the data in perfect first-normal form, and then Redis for storing difficult joins, caches, queues, etc. It would be a perfect marriage.
- dickeytk 15y ago
  YES! It's a winning powerhouse.
tkahn6 15y ago
In order to argue that redis isn't a database you have to first define what a database is and explain why redis doesn't fit that definition. All the author did was assert that redis isn't a database and give 3 use cases. This is just bad writing, to be blunt. And for what it's worth antirez wrote an HN clone using redis as the database.
nostrademons 15y ago
I'm curious why you would use Redis instead of just using in-memory datastructures in your app servers? It's trivially easy to implement a leaderboard as a priority queue, for example. And it eliminates the need to run yet another server and deal with the associated RPC & command parsing overhead. That's the approach Hacker News and Viaweb took, along with Mailinator and probably several other startups.
- jdunck 15y ago
  The idea is that you coordinate concurrent read/write to the same structures, yielding a shared data view without formal coordination. Also, the structures may be large enough that you don't want them all in-memory for each client.
  - nostrademons 15y ago
    I was kinda assuming that the system already has a separate appserver tier from the webserver tier (HN doesn't, but many other real-world apps would). Separating application logic from HTML formatting is usually a good idea, if only because HTML templating tends to be CPU-intensive but memory-light while application logic is often CPU-light but memory-intensive. That's an orthogonal issue, though - you can run as many appserver instances as are necessary for your dataset. I guess I was wondering why, in your app server, you don't just add a big in-process heap and use the normal language mechanisms to access it wherever you'd return your leaderboard info? The concurrency issue is interesting - how does Redis handle it? Does it have some sort of STM, or is it all because everything executes in a single thread in Redis? If it's the latter, you'd get that for free in a single-threaded appserver (although you probably don't want a single-threaded appserver).
    - tabbyjabby 15y ago
      Redis is single-threaded. Also, not everyone uses stateful app servers.
- antoncohen 15y ago
  > why you would use Redis instead of just using in-memory datastructures in your app servers? If you mean actually storing the data (scores on a leaderboard) on the app servers, the problem is it can only scale so far. If data/state is stored on app servers you can't load balance across multiple servers. HN runs on one server, and it has been hitting scalability issues lately. It's also harder to do high-availability, if everything runs on one server there's nothing to fail-over to.
jbwyme 15y ago
For some reason, people seem to think it's a key-value store, or some persistent database, but that's totally not it at all. From what I understand, it is actually a key-value store and is basically a superset of memcached. Therefore, if my understanding is true you could use it merely as a key-value store as well and use its other features (native sets, lists, etc, pub-sub, and persistance) as needed.
- dickeytk 15y ago
  Your understanding is correct, but those 'other features' is right where Redis becomes interesting. If you just want a better memcache, use membase.
Detrus 15y ago
so if you've been thinking it's persistent, get that OUT of your head now! On persistance from antirez http://antirez.com/post/short-term-redis-plans.html http://antirez.com/post/short-term-redis-plans.html We want also work both in the communication (most users don't understand that Redis with both AOF and RDB enabled is very durable already, and this is the setup we suggest) and the implementation to make sure that Redis AOF can be a very durable solution, as durable as the best SQL databases out there.
freeformz 15y ago
Hah! I've been saying this for a while. Good to see it on Hacker News!
jasonjei 15y ago
I use Redis as a distributed locking mechanism too, especially with its setex feature which can help reduce deadlocks. We have a UUID string as the value for the key, and the only way to release the lock is that the UUID must be passed and matched against the value in Redis.
antirez 15y ago
This post is not correct, but of course the biggest problems are the claiming that Redis is not persistent, and the use of KEYS command.

65 points by dickeytk 15y ago

Hmm. The author's recommendation to use Redis' 'KEYS' command to pattern match keys for invalidation is a dangerous one. 'KEYS' runs in O(n) time. If you're using Redis in a serious production environment, you do not want want to be using an O(n) command each time you need to invalidate a group of keys. It's better to group related items in a Redis hash table, and that way they are all stored under one common key. Honestly, I think this article is a more impoverished version of antirez's post on the same topic [1]. antirez, being one of the principal authors of Redis, is a much more authoritative source, and he actually describes all the patterns that this author described in greater detail. [1] http://antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html http://antirez.com/post/take-advantage-of-redis-adding-it-to...

LeafStorm 15y ago
And the O(N) is against your entire keyset, and not just the keys you're erasing. Which makes it even worse.
dickeytk 15y ago
Thanks for linking the antirez article, I actually haven't seen that one before I wrote the article. I put a link to it in the article. As far as they keys thing, it's obviously been a major point of contention with my article. I know the warning, I just don't think it's actually a problem for cache invalidation. See below on another comment of someone having similar concerns for my explanation why.
dickeytk 15y ago
tabby, I've also updated the article to mention the fact that KEYS may not be a solid use of the system. I would appreciate any feedback!
- tabbyjabby 15y ago
  Sorry, I didn't mean to come across as a dick. I'm glad that you've taken the feedback of your readers into consideration. I'm a big fan of Redis, and it's a key component of our stack. Sorted sets are useful for a lot more than just leader boards, though that is a good use case for them. It's a bit late here, so I'm not feeling up to writing a big post, but I'm considering writing my own blog post on my experiences with Redis.
  - dickeytk 15y ago
    I don't think you were at all. Many people mentioned the same thing you did. I'm new at blogging, so I'm trying to take in everything and produce the best content I can, so I love the feedback! I would love to see a follow-up article too by the way!

philjackson 15y ago

"It has some persistence support, but does not appear to be super durable. If you're thinking of using like that though, you're misunderstanding the tool." This warrants a much more detailed explanation. The author should have spoken about possible durability options, like the append-only file, which, given the right configuration, makes Redis "fully-durable" at the inevitable sacrifice of some speed. A more informative read would be: http://redis.io/topics/persistence http://redis.io/topics/persistence

typicalrunt 15y ago

Now, this is where Redis comes in. You can find keys on wildcards! So you can just query it like so: keys post/83/* No no no. This is slow and there is a large warning section in the notes (http://redis.io/commands/keys http://redis.io/commands/keys) about using this in production environments. [Edit: Why is this bad? Think about if you have millions of keys in your environment. KEYS will need to iterate over a million keys to find the ones that match your pattern.] As an alternative to using KEYS, Redis provides the hash object. Store everything abou an object in a hash (e.g.: "posts:83") and then just delete the hash object. Everything under it will be removed as well. If you need to know what's in the hash that is going to be deleted, use the HKEYS command (which has no such warning about performance).

dickeytk 15y ago
Right, I know what the docs say, however, it's fine to use this for cache invalidation (but maybe not other use-cases) in production. The docs also say: "While the time complexity for this operation is O(N), the constant times are fairly low. For example, Redis running on an entry level laptop can scan a 1 million key database in 40 milliseconds." If you have so many keys that this is an issue for cache invalidation, you should be using memcached anyways. (Since it can be distributed, where in Redis distribution is left up to you to figure out) I wasn't able to dig it up, but I know I read an article about some consulting group making a site for a major shoe company where they did exactly this. Your hash method isn't the best way either though since it's more efficient to store everything in individual key-value pairs, and hashes cannot be nested. Really the BEST way to do this in Redis is to use a set containing all they keys related to an object, then clear each of them out when destroying an item.
- dickeytk 15y ago
  It's no authoritative source, but I'm obviously not the only one that thinks this: http://www.webdevelopment-blog.com/2011/07/puma-on-redis/ http://www.webdevelopment-blog.com/2011/07/puma-on-redis/
  - tabbyjabby 15y ago

Mikushi 15y ago

Valid article, mostly. But the point is still valid, and i feel that most people using Redis do not get it, it is a data structure server, it's not that hard to understand... Use it as such and you won't be disappointed.

dickeytk 15y ago
Yes!! Thank you so much. That's exactly the point I was trying to make.

pkulak 15y ago

If I was greenfielding a new project tomorrow, I'd use a Redis/MySQL combo. MySQL for all the data in perfect first-normal form, and then Redis for storing difficult joins, caches, queues, etc. It would be a perfect marriage.

dickeytk 15y ago
YES! It's a winning powerhouse.

tkahn6 15y ago

In order to argue that redis isn't a database you have to first define what a database is and explain why redis doesn't fit that definition. All the author did was assert that redis isn't a database and give 3 use cases. This is just bad writing, to be blunt. And for what it's worth antirez wrote an HN clone using redis as the database.

nostrademons 15y ago

I'm curious why you would use Redis instead of just using in-memory datastructures in your app servers? It's trivially easy to implement a leaderboard as a priority queue, for example. And it eliminates the need to run yet another server and deal with the associated RPC & command parsing overhead. That's the approach Hacker News and Viaweb took, along with Mailinator and probably several other startups.

jdunck 15y ago
The idea is that you coordinate concurrent read/write to the same structures, yielding a shared data view without formal coordination. Also, the structures may be large enough that you don't want them all in-memory for each client.
- nostrademons 15y ago
  I was kinda assuming that the system already has a separate appserver tier from the webserver tier (HN doesn't, but many other real-world apps would). Separating application logic from HTML formatting is usually a good idea, if only because HTML templating tends to be CPU-intensive but memory-light while application logic is often CPU-light but memory-intensive. That's an orthogonal issue, though - you can run as many appserver instances as are necessary for your dataset. I guess I was wondering why, in your app server, you don't just add a big in-process heap and use the normal language mechanisms to access it wherever you'd return your leaderboard info? The concurrency issue is interesting - how does Redis handle it? Does it have some sort of STM, or is it all because everything executes in a single thread in Redis? If it's the latter, you'd get that for free in a single-threaded appserver (although you probably don't want a single-threaded appserver).
  - tabbyjabby 15y ago
    Redis is single-threaded. Also, not everyone uses stateful app servers.
antoncohen 15y ago
> why you would use Redis instead of just using in-memory datastructures in your app servers? If you mean actually storing the data (scores on a leaderboard) on the app servers, the problem is it can only scale so far. If data/state is stored on app servers you can't load balance across multiple servers. HN runs on one server, and it has been hitting scalability issues lately. It's also harder to do high-availability, if everything runs on one server there's nothing to fail-over to.

jbwyme 15y ago

For some reason, people seem to think it's a key-value store, or some persistent database, but that's totally not it at all. From what I understand, it is actually a key-value store and is basically a superset of memcached. Therefore, if my understanding is true you could use it merely as a key-value store as well and use its other features (native sets, lists, etc, pub-sub, and persistance) as needed.

dickeytk 15y ago
Your understanding is correct, but those 'other features' is right where Redis becomes interesting. If you just want a better memcache, use membase.

Detrus 15y ago

so if you've been thinking it's persistent, get that OUT of your head now! On persistance from antirez http://antirez.com/post/short-term-redis-plans.html http://antirez.com/post/short-term-redis-plans.html We want also work both in the communication (most users don't understand that Redis with both AOF and RDB enabled is very durable already, and this is the setup we suggest) and the implementation to make sure that Redis AOF can be a very durable solution, as durable as the best SQL databases out there.

freeformz 15y ago

Hah! I've been saying this for a while. Good to see it on Hacker News!

jasonjei 15y ago

I use Redis as a distributed locking mechanism too, especially with its setex feature which can help reduce deadlocks. We have a UUID string as the value for the key, and the only way to release the lock is that the UUID must be passed and matched against the value in Redis.

antirez 15y ago

This post is not correct, but of course the biggest problems are the claiming that Redis is not persistent, and the use of KEYS command.