3 ms·
A couple points I would make with respect to the article: - In-memory databases offer few advantages over a disk-backed database with a properly designed I/O s
by jandrewrogers 12y ago
A couple points I would make with respect to the article:
- In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. In-memory databases are generally only faster if the disk-backed database uses mmap() for cache replacement or similarly terrible I/O scheduling. The big advantage of in-memory databases is that you avoid the enormously complicated implementation task of writing a good I/O scheduler and disk cache. For the user, there is little performance difference for a given workload on a given piece of server hardware.
- Data structure and algorithms have long existed for supercomputing applications that are very effective at exploiting cache and RAM locality. Most supercomputing applications are actually bottlenecked by memory bandwidth (not compute). Few databases do things this way -- it is a bit outside the evolutionary history of database internals -- because few database designers have experience optimizing for memory bandwidth. This is one of the reasons that some disk-backed databases like SpaceCurve have much higher throughput than in-memory databases: excellent I/O scheduling (no I/O bottlenecks) and memory bandwidth optimized internals (higher throughput of what is in cache).
The trend in database engines is highly pipelined execution paths within a single thread with almost no coordination or interactions between threads. If you look at codes that are designed to optimize memory bandwidth, this is the way they are designed. No context switching and virtually no shared data structures. Properly implemented, you can easily saturate both sides of a 10GbE NIC on a modest server simultaneously for many database workloads.
- toomim 12y agoCan you give some refs to make these statements more real? I'm trying to understand whether an in-memory db would be faster than postgres, for instance. What's the postgres I/O scheduler, and is it good? Are there benchmarks somewhere showing the difference? Why does the I/O scheduler make a difference for in-memory vs. disk databases? Is this a subsystem that caches the database in memory? Are you saying that, with proper caching, a disk-based database is as efficient as an in-memory db?
- jandrewrogers 12y agoPostgreSQL is a bit of a hybrid in terms of scheduling. It has its own disk cache but still goes through the kernel mechanisms. Classic database I/O schedulers are not portable so it is a bit of a challenge to implement one in big open source projects for reasons outside of technical ability. With proper cache and I/O scheduler, the same workload will fit in memory, so the only way the disk gets in the way is if the I/O scheduling does something suboptimal with respect to writes (which happens a lot with the kernel caching behavior). Modern database servers typically have more disk bandwidth than network bandwidth and therefore most write workloads should be able to go through storage at the network's wire speed, at least in theory. In practice, I/O scheduling behavior from memory to storage tends to be bursty or poorly timed. Consequently, the instantaneous I/O bandwidth requirements can exceed the effective disk bandwidth for brief periods and performance degrades. A really good database I/O scheduler, cache, and execution engine work together to basically makes sure that peak bandwidth demands to the disk subsystem are never much worse than the network wire speed. Achieving this is not trivial and requires a lot of clever dynamic resource optimization but many sophisticated database engines implement this to some degree or another.
- jbergens 12y agoI think it also depends on how you structure your database and how you use it. For some things a NOSQL design is great and most of them don't have to lock over multiple tables they might be very fast at getting structured records. They might be terrible if you try to join different records (especially if they are located on different shards) and some don't have ACID if that is a requirement. Some databases like Datomic stores information in a completely different way that may be very fast for some use cases. If you have an application or a specific use case you can try a few different databases and measure how they work. I tried a small memory database and got much better performance than SQL Server gave me while the queries were easier to write, for other things this database would not have worked at all or it might have been about as fast as a relational db. As a general rule the memory db providers seems to make the case that they are faster than normal relational databases. Take a look at VoltDb or StarCounter for example.
- mpweiher 12y ago> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. Hmm...Michael Stonebraker would probably disagree with you on that. For his different newer projects, he claims and somewhat convincingly shows 10x or more performance improvements. The analysis is that ~10% of a typical disk-based system is useful work. By radically simplifying as a result of doing away with the disk-store, you remove those overheads. A similar argument (also with benchmarks) is made by the RAMCloud people, they claim up to 1000x perf. increase over disk-based storage for a data-center. Since I am mostly a dabbler (but have also managed to outperform RDBMs by factor of 1000 or more using RAM-based techniques), I would be curious as to what these papers get wrong. http://downloads.voltdb.com/datasheets_collateral/technical_overview.pdf http://downloads.voltdb.com/datasheets_collateral/technical_... http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud
- platz 12y agoIt's not just the disk store, also avoiding the time spent locking and unlocking in row-based architectures (i.e. using lock-free algorithms) http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/ http://www.se-radio.net/2013/12/episode-199-michael-stonebra...
- mpweiher 12y ago"...single-threaded, lock-free, doesn’t require disk I/O in the critical path, ..." IIRC, the single-threadedness is made possible in large part by not having I/O on the critical path. (When you wait on I/O, a single threaded design stalls completely, so either you multi-thread somehow or your throughput dies). :-)
- jandrewrogers 12y agoNo, Stonebraker is making many architectural assumptions that do not necessarily hold. Some observations about the design of good disk-based systems: - Typical inexpensive server and disk systems today are quite a bit different than even five years ago. A database internals design heuristic that would be valid several years ago no longer applies in many cases. An "optimal" design can vary widely due to relatively small changes to the assumed constraints. - The disk bandwidth typically exceeds network bandwidth by a significant degree. Even if you are saturating the network with inserts and similar, in theory you should be able to drive that through storage if you use the available IOPS efficiently and proactively. - RAM is RAM. Whether backed by disk or not, the amount of data it can hold is approximately the same. Any significant differences in what you can put in RAM between in-memory and on-disk architectures has more to do with internal implementation and design choices, it is not intrinsic. - In modern database kernels, no thread is waiting for I/O operations to complete. There is also few or no tree structures, locking, secondary indexing, context switching, and similar which are frequently the source of poor throughput. It tends to be much more linear, pipelined, and parallel. All of the above aside, regardless of design, a database engine only needs enough throughput to saturate the network interface under load. Once the network is saturated, nothing you do in the database engine will improve the effective throughput of the database. It turns out that it is pretty straightforward to design a disk-backed database kernel that can saturate 10GbE full-duplex for a wide range of workloads. Consequently, the supposed performance benefits of in-memory are largely moot. It simplifies implementation but does not address a real performance problem on modern hardware relative to a modern database kernel design.
- ehsanu1 12y agoWhat's so terrible about mmap()? LMDB seems to use it to great effect: http://symas.com/mdb/ http://symas.com/mdb/
- ddorian43 12y agolmdb is single-writter-thread
- hyc_symas 12y agoHow is that in any way relevant? All ARIES write-ahead-logger systems are single-writer to their logs too. No matter what kind of BS they sell you about high concurrency with fine-grained locking, it all funnels thru that single bottleneck. The fact is that LMDB's single-writer design is not a concurrency liability in real workloads. http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf http://symas.com/mdb/#bench http://symas.com/mdb/#bench