6 ms·

Is it true that Redis requires that the whole dataset fit in RAM? I've seen that mentioned a few times and if so, it's a serious consideration/fact to know abo

Is it true that Redis requires that the whole dataset fit in RAM? I've seen that mentioned a few times and if so, it's a serious consideration/fact to know about when deploying apps.

antirez 17y ago
Absolutely true. This is why Redis can do things that with an on-disk storage engine are almost impossible. This our reasoning: in order to really scale, with every kind of DB, if data access is evenly distributed like happens in many web 2.0 apps, the DB must fit in memory anyway (being this OS disk cache, or directly used memory like in Redis). What really matters is persistence and replication. Asynchronous disk dumps gives us all this: data is persistent, replication is fast and non blocking and trivial to setup. And because everything is in RAM we are free to do high level stuff without to care about how data is stored on disk. On disk we have just a "stupid" dump that we reload a startup. In some way this is what Paul Graham described in some comment years ago: take all the data in memory in your application, and just write a file in append-mode to reload it later. Basically this is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Redis :)
- spoiledtechie 17y ago
  Ya but what happens when Solid state hard drives come into play. And they will be soon. I, a asp.net guy will be able to compete with Redis then.
  - antirez 17y ago
    If RAM gets so cheap that Solid state hard drivers will come into play you'll have your Linux box with 512 GB of Ram by default. Btw I'm happy with ASP guys competing with Redis :) Also if someone stole what's good about Redis and creates FooBaredDB that is just better I'll drop Redis and use FooBaredDB.
- pierrefar 17y ago
  Interesting. Thanks for the detailed explanation! I'll be checking it out in detail (hopefully) soon.
- petercooper 17y ago
  On disk we have just a "stupid" dump that we reload a startup. In a lazy fashion, or not? That is, if I have a 10GB data set, will I be either waiting X minutes for a basic restart or dealing with sluggish queries for a slightly lesser time? (Update for anyone who wants an answer too: I dug around the docs and found that it's a full read into memory on restart, not lazy loading. They give an example of "It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID." So if frequent restarts are in your plan for some reason - not that they should be, of course - it might not be a goer.)

Is it true that Redis requires that the whole dataset fit in RAM? I've seen that mentioned a few times and if so, it's a serious consideration/fact to know about when deploying apps.

antirez 17y ago

Absolutely true. This is why Redis can do things that with an on-disk storage engine are almost impossible. This our reasoning: in order to really scale, with every kind of DB, if data access is evenly distributed like happens in many web 2.0 apps, the DB must fit in memory anyway (being this OS disk cache, or directly used memory like in Redis). What really matters is persistence and replication. Asynchronous disk dumps gives us all this: data is persistent, replication is fast and non blocking and trivial to setup. And because everything is in RAM we are free to do high level stuff without to care about how data is stored on disk. On disk we have just a "stupid" dump that we reload a startup. In some way this is what Paul Graham described in some comment years ago: take all the data in memory in your application, and just write a file in append-mode to reload it later. Basically this is an ad hoc, informally-specified, bug-ridden, slow implementation of half of Redis :)

spoiledtechie 17y ago
Ya but what happens when Solid state hard drives come into play. And they will be soon. I, a asp.net guy will be able to compete with Redis then.
- antirez 17y ago
  If RAM gets so cheap that Solid state hard drivers will come into play you'll have your Linux box with 512 GB of Ram by default. Btw I'm happy with ASP guys competing with Redis :) Also if someone stole what's good about Redis and creates FooBaredDB that is just better I'll drop Redis and use FooBaredDB.
pierrefar 17y ago
Interesting. Thanks for the detailed explanation! I'll be checking it out in detail (hopefully) soon.
petercooper 17y ago
On disk we have just a "stupid" dump that we reload a startup. In a lazy fashion, or not? That is, if I have a 10GB data set, will I be either waiting X minutes for a basic restart or dealing with sluggish queries for a slightly lesser time? (Update for anyone who wants an answer too: I dug around the docs and found that it's a full read into memory on restart, not lazy loading. They give an example of "It takes about 45 seconds to restore a 2 GB database on a fairly standard system, no RAID." So if frequent restarts are in your plan for some reason - not that they should be, of course - it might not be a goer.)