4 ms·

While reading people talking about writing data to permanent store, I got an interesting (well at least for me!) question. My initial idea was that it would be

by gedrap 12y ago

While reading people talking about writing data to permanent store, I got an interesting (well at least for me!) question. My initial idea was that it would be possible to store data to be written at RAM at first, and periodically flush to hard drive / DB. But then on other hand, OS does that already by using cache and flushing it, just at much lower level. Some DBs are probably doing it too. So the question is, is it worth implementing strategies like that (home made cache), or it is a better idea to trust OS/DB by default?

jimktrains2 12y ago
> My initial idea was that it would be possible to store data to be written at RAM at first, and periodically flush to hard drive / DB. Sure, but you lose anything that isn't written to disk. IIRC you can set up Mongo and Redis to do this (I'm not familiar with many of the other options). At this point you're no longer an ACID database. Most DBs will cache as much as they possibly can in memory. Often index's will take priority, but rows that are being accessed fairly frequently will also be in their cache. Obviously the details are different per db and configuration, but in general, it gets cached. Also, when you fwrite your data actually sits in RAM for a bit before being written to disk. Databases and other applications that actually require data to be on the disk right now need to call fsync. (Aside, even then you're disk can and will store writes in it's own internal buffer before it can be written to disk. The OS doesn't have much control over this.) A lot of people will, to avoid having to spin up db replicas, cache rows in redis or memcache and use those for reads whenever possible. It has its advantages and disadvantages, but for pk-based reads you can decrease the number of operations your DB needs to do at once (allowing it, for example, to focus on writes.
lenn0x 12y ago
I can speak from experience on this, home made cache can be much better depending on your avg object sizes. Letting the OS make decisions usually in databases is the wrong choice. The data is cached at page level (4k), if you have many objects randomly spread across your working set it's not going to be as efficient as object caching. Usually the more control you desire the better for the performance of your database. One technique that is used if you want to just let the OS deal with cache/paging of data is you have a dedicated commit log that is written with fsync on every update or periodically depending on your durability requirements. The OS usually does a pretty good job, but remember it's designed to be generic not for a specific use case. When you really care about performance of a database you tend to bypass what the OS does, and take control bypassing things like the scheduler, cache, etc.
mbesto 12y ago
SAP HANA (and then to OCFS or GPFS) does this: http://en.wikipedia.org/wiki/SAP_HANA http://en.wikipedia.org/wiki/SAP_HANA And AFAIK, InnoDB (then to MySQL) does as well.
rakoo 12y ago
Trust the OS, and don't use 70's technique like separating RAM and disk. Here's a really interesting piece on how PHK built varnish by using "modern" software techniques: https://www.varnish-cache.org/trac/wiki/ArchitectNotes https://www.varnish-cache.org/trac/wiki/ArchitectNotes And here's a Key-Value db you can use in your own programs that reuses the same idea, ie ask the OS for some space, trust it for RAM/Disk allocation; it turns out it is excellent at what it does thanks to careful design: http://symas.com/mdb/ http://symas.com/mdb/

by gedrap 12y ago

jimktrains2 12y ago

> My initial idea was that it would be possible to store data to be written at RAM at first, and periodically flush to hard drive / DB. Sure, but you lose anything that isn't written to disk. IIRC you can set up Mongo and Redis to do this (I'm not familiar with many of the other options). At this point you're no longer an ACID database. Most DBs will cache as much as they possibly can in memory. Often index's will take priority, but rows that are being accessed fairly frequently will also be in their cache. Obviously the details are different per db and configuration, but in general, it gets cached. Also, when you fwrite your data actually sits in RAM for a bit before being written to disk. Databases and other applications that actually require data to be on the disk right now need to call fsync. (Aside, even then you're disk can and will store writes in it's own internal buffer before it can be written to disk. The OS doesn't have much control over this.) A lot of people will, to avoid having to spin up db replicas, cache rows in redis or memcache and use those for reads whenever possible. It has its advantages and disadvantages, but for pk-based reads you can decrease the number of operations your DB needs to do at once (allowing it, for example, to focus on writes.

lenn0x 12y ago

I can speak from experience on this, home made cache can be much better depending on your avg object sizes. Letting the OS make decisions usually in databases is the wrong choice. The data is cached at page level (4k), if you have many objects randomly spread across your working set it's not going to be as efficient as object caching. Usually the more control you desire the better for the performance of your database. One technique that is used if you want to just let the OS deal with cache/paging of data is you have a dedicated commit log that is written with fsync on every update or periodically depending on your durability requirements. The OS usually does a pretty good job, but remember it's designed to be generic not for a specific use case. When you really care about performance of a database you tend to bypass what the OS does, and take control bypassing things like the scheduler, cache, etc.

mbesto 12y ago

SAP HANA (and then to OCFS or GPFS) does this: http://en.wikipedia.org/wiki/SAP_HANA http://en.wikipedia.org/wiki/SAP_HANA And AFAIK, InnoDB (then to MySQL) does as well.

rakoo 12y ago

Trust the OS, and don't use 70's technique like separating RAM and disk. Here's a really interesting piece on how PHK built varnish by using "modern" software techniques: https://www.varnish-cache.org/trac/wiki/ArchitectNotes https://www.varnish-cache.org/trac/wiki/ArchitectNotes And here's a Key-Value db you can use in your own programs that reuses the same idea, ie ask the OS for some space, trust it for RAM/Disk allocation; it turns out it is excellent at what it does thanks to careful design: http://symas.com/mdb/ http://symas.com/mdb/