6 ms·

Redis 1.1 supports durability via append only file

55 points by antirez 17y ago

antirez 17y ago
I want to share this with you as I think it's a game changer for everybody that looked at Redis, didn't liked the snapshotting because their data is important, and decided to discard it from the viable alternatives. Well I also hope the article is somewhat informative as a side effect. p.s. version 1.1 is currently in beta, the feature is available on Git, a stable version (rc1) will be released at the end of this year.
- pierrefar 17y ago
  Will the client libraries be updated in tandem with the 1.1 release? Especially the PHP extension as it seems to be missing SETNX as far as I can tell.
  - antirez 17y ago
    Hello pierrefar. Yes, the most important client libs will support (many already do) 1.1 sorted sets, MSET and MSETNX commands (Multi keys set in a single command atomically), and the other features once 1.1 will be released. But about PHP, I've good news, there are two new fully featured high quality implementations of the Redis protocol for PHP: Predis (http://github.com/nrk/predis/ http://github.com/nrk/predis/) and Rediska (http://rediska.geometria-lab.net/ http://rediska.geometria-lab.net/). Both with support for consistent hashing. Also the PHP C module got two new developers and is now much more stable, supporting the full 1.0 protocol AFAIK: http://github.com/owlient/phpredis http://github.com/owlient/phpredis So the client libs arena is getting better and better fortunately. Other good quality client libs are for Ruby, Java, and Go Language. Python is getting better with the time too.
    - pierrefar 17y ago
      Thanks. I'm actually writing a CMS using Redis and absolutely need SETNX (which one current PHP-only lib supports well). I would like to move to a compiled PHP module so that I get a nice performance boost. Coupled with this new log-based persistence, I'll be more comfortable with the whole set up. If you want to talk more, my email is hello at (my username).com.
mahmud 17y ago
And the fun continues! Gonna upgrade to latest and start supporting 1.1 API in cl-redis tomorrow. Win32 builds are greatly appreciated. Contacted the OHM developer yesterday in #ohm on freenode; will have a chat and see if we can make that into a language-independent protocol (I have been making the same thing for Common Lisp for sometime now.)
- antirez 17y ago
  Great, you can find me on #Redis (freenode) as well if I can help. About Win32: Redis compiles without problems on Cygwin but at least two tests are failing (it's about INCR with 64 bit values, strtol() is not working well maybe?).
- soveran 17y ago
  Great idea, let's talk about it on IRC.
moe 17y ago
Sad to see that a redis database is still limited by the available RAM. This seems like a bigger showstopper than durability...
- antirez 17y ago
  Hello moe, actually a lot of people overestimate the amount of their data. Btw there are a few solutions: * given that RAM is cheap, use a Linux box with 32 or 64 GB of RAM. After all with the performances of Redis you need a single box when with other solutions more than one server is needed. It's hard to saturate 150,000 write or read per second even with a lot of users. * to split data across different servers (using application-level partitioning or consistent hashing). * to wait for Redis VM implementation (virtual memory). Basically there is the plan to implement something like Operating Systems already do with memory pages. More information (and why we can't just let the OS do the work for us) in the Redis FAQ at http://code.google.com/p/redis/wiki/FAQ http://code.google.com/p/redis/wiki/FAQ Search for "Do you plan to implement Virtual Memory in Redis? Why don't just let the Operating System handle it for you?".
  - paraschopra 17y ago
    Hi Antirez, >to wait for Redis VM implementation (virtual memory) When can we expect this?
    - antirez 17y ago
      It's something like two release away. The next release will implement the Hash type, needed for virtual memory so that applications can be designed to put data that is accessed with similar patterns in the same hash. The release after this VM will be implemented. It's something like <= 6 months.
      - vdm 17y ago
        Wow! It's not every day an open source developer gives an estimate for a future feature. I'm impressed.
jbellis 17y ago
what is the speed hit?
- antirez 17y ago
  read performance is the same of course, as the append only file is not touched. write performances from the first tests appear to be like 70% of snapshotting (tested with redis-benchmark), but more tests are needed to really understand what's the write speed hit.
  - z8000 17y ago
    Are you putting the journal on its own disk?
    - antirez 17y ago
      No I'm just trying it in my macbook pro :)
julien 17y ago
Hum, that Reddis thingy really starts to be more and more exciting. We can't wait to implement it at Superfeedr :)
- mahmud 17y ago
  We use it as a log server, a message queue, and as a mailbox cache (inter-user messages without attachments.) It's a beast.

55 points by antirez 17y ago

I want to share this with you as I think it's a game changer for everybody that looked at Redis, didn't liked the snapshotting because their data is important, and decided to discard it from the viable alternatives. Well I also hope the article is somewhat informative as a side effect. p.s. version 1.1 is currently in beta, the feature is available on Git, a stable version (rc1) will be released at the end of this year.

pierrefar 17y ago
Will the client libraries be updated in tandem with the 1.1 release? Especially the PHP extension as it seems to be missing SETNX as far as I can tell.
- antirez 17y ago
  Hello pierrefar. Yes, the most important client libs will support (many already do) 1.1 sorted sets, MSET and MSETNX commands (Multi keys set in a single command atomically), and the other features once 1.1 will be released. But about PHP, I've good news, there are two new fully featured high quality implementations of the Redis protocol for PHP: Predis (http://github.com/nrk/predis/ http://github.com/nrk/predis/) and Rediska (http://rediska.geometria-lab.net/ http://rediska.geometria-lab.net/). Both with support for consistent hashing. Also the PHP C module got two new developers and is now much more stable, supporting the full 1.0 protocol AFAIK: http://github.com/owlient/phpredis http://github.com/owlient/phpredis So the client libs arena is getting better and better fortunately. Other good quality client libs are for Ruby, Java, and Go Language. Python is getting better with the time too.
  - pierrefar 17y ago
    Thanks. I'm actually writing a CMS using Redis and absolutely need SETNX (which one current PHP-only lib supports well). I would like to move to a compiled PHP module so that I get a nice performance boost. Coupled with this new log-based persistence, I'll be more comfortable with the whole set up. If you want to talk more, my email is hello at (my username).com.

mahmud 17y ago

And the fun continues! Gonna upgrade to latest and start supporting 1.1 API in cl-redis tomorrow. Win32 builds are greatly appreciated. Contacted the OHM developer yesterday in #ohm on freenode; will have a chat and see if we can make that into a language-independent protocol (I have been making the same thing for Common Lisp for sometime now.)

antirez 17y ago
Great, you can find me on #Redis (freenode) as well if I can help. About Win32: Redis compiles without problems on Cygwin but at least two tests are failing (it's about INCR with 64 bit values, strtol() is not working well maybe?).
soveran 17y ago
Great idea, let's talk about it on IRC.

moe 17y ago

Sad to see that a redis database is still limited by the available RAM. This seems like a bigger showstopper than durability...

antirez 17y ago
Hello moe, actually a lot of people overestimate the amount of their data. Btw there are a few solutions: * given that RAM is cheap, use a Linux box with 32 or 64 GB of RAM. After all with the performances of Redis you need a single box when with other solutions more than one server is needed. It's hard to saturate 150,000 write or read per second even with a lot of users. * to split data across different servers (using application-level partitioning or consistent hashing). * to wait for Redis VM implementation (virtual memory). Basically there is the plan to implement something like Operating Systems already do with memory pages. More information (and why we can't just let the OS do the work for us) in the Redis FAQ at http://code.google.com/p/redis/wiki/FAQ http://code.google.com/p/redis/wiki/FAQ Search for "Do you plan to implement Virtual Memory in Redis? Why don't just let the Operating System handle it for you?".
- paraschopra 17y ago
  Hi Antirez, >to wait for Redis VM implementation (virtual memory) When can we expect this?
  - antirez 17y ago
    It's something like two release away. The next release will implement the Hash type, needed for virtual memory so that applications can be designed to put data that is accessed with similar patterns in the same hash. The release after this VM will be implemented. It's something like <= 6 months.
    - vdm 17y ago
      Wow! It's not every day an open source developer gives an estimate for a future feature. I'm impressed.

jbellis 17y ago

what is the speed hit?

antirez 17y ago
read performance is the same of course, as the append only file is not touched. write performances from the first tests appear to be like 70% of snapshotting (tested with redis-benchmark), but more tests are needed to really understand what's the write speed hit.
- z8000 17y ago
  Are you putting the journal on its own disk?
  - antirez 17y ago
    No I'm just trying it in my macbook pro :)

julien 17y ago

Hum, that Reddis thingy really starts to be more and more exciting. We can't wait to implement it at Superfeedr :)

mahmud 17y ago
We use it as a log server, a message queue, and as a mailbox cache (inter-user messages without attachments.) It's a beast.