3 ms·
> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. Hmm...Michael Stonebraker would probably disagre
by mpweiher 12y ago
> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler.
Hmm...Michael Stonebraker would probably disagree with you on that. For his different newer projects, he claims and somewhat convincingly shows 10x or more performance improvements. The analysis is that ~10% of a typical disk-based system is useful work. By radically simplifying as a result of doing away with the disk-store, you remove those overheads.
A similar argument (also with benchmarks) is made by the RAMCloud people, they claim up to 1000x perf. increase over disk-based storage for a data-center.
Since I am mostly a dabbler (but have also managed to outperform RDBMs by factor of 1000 or more using RAM-based techniques), I would be curious as to what these papers get wrong.
http://downloads.voltdb.com/datasheets_collateral/technical_overview.pdf http://downloads.voltdb.com/datasheets_collateral/technical_...
http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf
https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud
- platz 12y agoIt's not just the disk store, also avoiding the time spent locking and unlocking in row-based architectures (i.e. using lock-free algorithms) http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/ http://www.se-radio.net/2013/12/episode-199-michael-stonebra...
- mpweiher 12y ago"...single-threaded, lock-free, doesn’t require disk I/O in the critical path, ..." IIRC, the single-threadedness is made possible in large part by not having I/O on the critical path. (When you wait on I/O, a single threaded design stalls completely, so either you multi-thread somehow or your throughput dies). :-)
- jandrewrogers 12y agoNo, Stonebraker is making many architectural assumptions that do not necessarily hold. Some observations about the design of good disk-based systems: - Typical inexpensive server and disk systems today are quite a bit different than even five years ago. A database internals design heuristic that would be valid several years ago no longer applies in many cases. An "optimal" design can vary widely due to relatively small changes to the assumed constraints. - The disk bandwidth typically exceeds network bandwidth by a significant degree. Even if you are saturating the network with inserts and similar, in theory you should be able to drive that through storage if you use the available IOPS efficiently and proactively. - RAM is RAM. Whether backed by disk or not, the amount of data it can hold is approximately the same. Any significant differences in what you can put in RAM between in-memory and on-disk architectures has more to do with internal implementation and design choices, it is not intrinsic. - In modern database kernels, no thread is waiting for I/O operations to complete. There is also few or no tree structures, locking, secondary indexing, context switching, and similar which are frequently the source of poor throughput. It tends to be much more linear, pipelined, and parallel. All of the above aside, regardless of design, a database engine only needs enough throughput to saturate the network interface under load. Once the network is saturated, nothing you do in the database engine will improve the effective throughput of the database. It turns out that it is pretty straightforward to design a disk-backed database kernel that can saturate 10GbE full-duplex for a wide range of workloads. Consequently, the supposed performance benefits of in-memory are largely moot. It simplifies implementation but does not address a real performance problem on modern hardware relative to a modern database kernel design.