4 ms·

When to Redis

39 points by transmit101 17y ago

antirez 17y ago
Related: Redis 1.1 (currently in beta but ready for some test) implements a sorted type supporting range queries. It's pretty useful when the need is to take million of items sorted with very fast top-N items operations or items with score between min and max, especially in presence of continuous updates to the items scores. If you are interested check http://code.google.com/p/redis/wiki/CommandReference http://code.google.com/p/redis/wiki/CommandReference (All the commands prefixed with "Z" are about the Sorted Set data type). Btw both insert / remove / and update-score operations are O(Log(N)). Top-N items is O(N) since the list of items is already sorted. Redis 1.1 will be released as stable in one or two months at max. For data structures and algorithms geeks: the sorted data type of Redis is implemented with a skip list, hacked to actually build a doubly-liked list, with the backward pointers being only at level 0 (we have reverse range operations, ZREVRANGE). Every element is also taken inside an hash table, so that it's possible to update the score of an element already inside the sorted set in O(Log(N)) time because we can get the old score in O(1) from the hash table, and then find the element in the skip list. Even when the scores are not distributed but there are large clusters of elements with the same score, the update is still O(Log(N)) thanks to some interesting trick.
almost 17y ago
Redis is really cool, I can think of lots of uses for fast and lightweight and there's at least one project where it would have saved me a lot of effort if it had existed then! Watch out for the Python library though! It doesn't even try to sanitize key names so malformed names cause all sorts of problems (including executing arbitrary commands). Plus unicode strings with non-ascii chars cause it to blow up. All very easy to fix of course, as soon as I get a chance I'll submit a patch if it hasn't already been done by someone else by then.
- antirez 17y ago
  Hello almost, I don't know Python so I'm not sure about this problems, btw starting from Redis 1.1 the client libraries will no longer be included in the main project (now they are in the tarball of Redis), and I'll try to stimulate much more a Bazar development style for client libraries. Btw the Ruby client lib is solid, like it appears to be the Java one. Still there are client libs that absolutely need to be improved. Currently the Ruby one is as far as I know the only one supporting consistent hashing but probably this problem will be fixed with the introduction of a new daemon 'redis-cluster' that will work as a proxy taking care to deal with the hash ring in a transparent way.
- deleted 17y ago
  [deleted]
- bham 17y ago
  Please submit a few reproducible cases to the mailing list. I'm sure Ludo or I (or you) could even patch it quickly. Thanks!
  - almost 17y ago
    I've committed a quick fix for the problems to me clone of the Git repo here: http://github.com/almost/redis http://github.com/almost/redis Check the changed doctests for examples of why it was a bad idea not to sanitize key names and other params. Ignore the changed doctests involving Decimals, I had to change those to get them to pass on my machine but they're not related to me fixes.
transmit101 17y ago
Redis is particularly useful for implementing simple message queueing systems where the full functionality or power of something like RabbitMQ is not required. For example, when parsing incoming data into a database, push incoming data onto one end of a Redis list, and have a seperate worker process popping them off the end and into CouchDB, MySQL or whatever. The fact that Redis list operations are atomic mean that multiple worker processes (possibly on seperate physical boxes) can process jobs simultaneously. Result: the internet is entirely decoupled from both the hard drive and the database, and extremely high performance is possible. Twitter Streaming API, anybody?..
chubbard 17y ago
Is Redis stable enough to replace using a database as persistent storage. There seems to be some indication that it is, but then there are paragraphs in the documentation that makes it seem like it's not. Particularly if your database size exceeds the machine's memory then you could lose data. Is one of the goals of the project to replace the need to use a database? Or will it always be more suitable cache or worker queue replacement?
- antirez 17y ago
  Hello, the documentation should be updated indeed. Now Redis supports the maxmemory directive that will start to reply with an error on write commands when the database is using more than the specified bytes of RAM.
tptacek 17y ago
There was a post earlier in the week about why Redis was "awesome" and a "lifestyle" and I posted a kind of snarky comment about why posts like that scare me away from Redis. After this post, I had an hourlong discussion with my lead dev on swapping out memcached for Redis so we could use it for our job scheduler. This post makes an excellent, compelling case for Redis. Just an interesting example of how two different evangelistic approaches can play out. I don't know if I'm representative of "normal people", though. (Don't say it.)
- roidrage 17y ago
  It was that snarky comment on my first post on "awesome" and "lifestyle" (although I thought I'd make pretty clear that I don't consider Redis a lifestyle) that made me write this second one. I'm glad it helped clarify some things.
- silentbicycle 17y ago
  Indeed. I went to a CouchDB presentation at a local conference, and afterward I asked about some transaction-related details. The presenter didn't understand, but kept repeating that the technology behind it was "way sexy". It was useless and off-putting.
  - janl 17y ago
    Which conference was that?
  - janl 17y ago
    Which conference was that?
    - silentbicycle 17y ago
      A BarCamp in Grand Rapids, MI, US.

39 points by transmit101 17y ago

Related: Redis 1.1 (currently in beta but ready for some test) implements a sorted type supporting range queries. It's pretty useful when the need is to take million of items sorted with very fast top-N items operations or items with score between min and max, especially in presence of continuous updates to the items scores. If you are interested check http://code.google.com/p/redis/wiki/CommandReference http://code.google.com/p/redis/wiki/CommandReference (All the commands prefixed with "Z" are about the Sorted Set data type). Btw both insert / remove / and update-score operations are O(Log(N)). Top-N items is O(N) since the list of items is already sorted. Redis 1.1 will be released as stable in one or two months at max. For data structures and algorithms geeks: the sorted data type of Redis is implemented with a skip list, hacked to actually build a doubly-liked list, with the backward pointers being only at level 0 (we have reverse range operations, ZREVRANGE). Every element is also taken inside an hash table, so that it's possible to update the score of an element already inside the sorted set in O(Log(N)) time because we can get the old score in O(1) from the hash table, and then find the element in the skip list. Even when the scores are not distributed but there are large clusters of elements with the same score, the update is still O(Log(N)) thanks to some interesting trick.

almost 17y ago

Redis is really cool, I can think of lots of uses for fast and lightweight and there's at least one project where it would have saved me a lot of effort if it had existed then! Watch out for the Python library though! It doesn't even try to sanitize key names so malformed names cause all sorts of problems (including executing arbitrary commands). Plus unicode strings with non-ascii chars cause it to blow up. All very easy to fix of course, as soon as I get a chance I'll submit a patch if it hasn't already been done by someone else by then.

antirez 17y ago
Hello almost, I don't know Python so I'm not sure about this problems, btw starting from Redis 1.1 the client libraries will no longer be included in the main project (now they are in the tarball of Redis), and I'll try to stimulate much more a Bazar development style for client libraries. Btw the Ruby client lib is solid, like it appears to be the Java one. Still there are client libs that absolutely need to be improved. Currently the Ruby one is as far as I know the only one supporting consistent hashing but probably this problem will be fixed with the introduction of a new daemon 'redis-cluster' that will work as a proxy taking care to deal with the hash ring in a transparent way.
deleted 17y ago
[deleted]
bham 17y ago
Please submit a few reproducible cases to the mailing list. I'm sure Ludo or I (or you) could even patch it quickly. Thanks!
- almost 17y ago
  I've committed a quick fix for the problems to me clone of the Git repo here: http://github.com/almost/redis http://github.com/almost/redis Check the changed doctests for examples of why it was a bad idea not to sanitize key names and other params. Ignore the changed doctests involving Decimals, I had to change those to get them to pass on my machine but they're not related to me fixes.

transmit101 17y ago

Redis is particularly useful for implementing simple message queueing systems where the full functionality or power of something like RabbitMQ is not required. For example, when parsing incoming data into a database, push incoming data onto one end of a Redis list, and have a seperate worker process popping them off the end and into CouchDB, MySQL or whatever. The fact that Redis list operations are atomic mean that multiple worker processes (possibly on seperate physical boxes) can process jobs simultaneously. Result: the internet is entirely decoupled from both the hard drive and the database, and extremely high performance is possible. Twitter Streaming API, anybody?..

chubbard 17y ago

Is Redis stable enough to replace using a database as persistent storage. There seems to be some indication that it is, but then there are paragraphs in the documentation that makes it seem like it's not. Particularly if your database size exceeds the machine's memory then you could lose data. Is one of the goals of the project to replace the need to use a database? Or will it always be more suitable cache or worker queue replacement?

antirez 17y ago
Hello, the documentation should be updated indeed. Now Redis supports the maxmemory directive that will start to reply with an error on write commands when the database is using more than the specified bytes of RAM.

tptacek 17y ago

There was a post earlier in the week about why Redis was "awesome" and a "lifestyle" and I posted a kind of snarky comment about why posts like that scare me away from Redis. After this post, I had an hourlong discussion with my lead dev on swapping out memcached for Redis so we could use it for our job scheduler. This post makes an excellent, compelling case for Redis. Just an interesting example of how two different evangelistic approaches can play out. I don't know if I'm representative of "normal people", though. (Don't say it.)

roidrage 17y ago
It was that snarky comment on my first post on "awesome" and "lifestyle" (although I thought I'd make pretty clear that I don't consider Redis a lifestyle) that made me write this second one. I'm glad it helped clarify some things.
silentbicycle 17y ago
Indeed. I went to a CouchDB presentation at a local conference, and afterward I asked about some transaction-related details. The presenter didn't understand, but kept repeating that the technology behind it was "way sexy". It was useless and off-putting.
- janl 17y ago
  Which conference was that?
- janl 17y ago
  Which conference was that?
  - silentbicycle 17y ago
    A BarCamp in Grand Rapids, MI, US.