11 ms·
BerkeleyDB is not so fast actually comparing to modern alternatives http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/
by snaky 14y ago
BerkeleyDB is not so fast actually comparing to modern alternatives
http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/
- jules 14y agoLies, damned lies, statistics, and benchmarks. Those benchmarks seem too good to be true. And reading the associated information they do look like that they are too good to be true. They claim zero-copy access to the database, which is great. But that probably means that in their read benchmarks, they are just returning a pointer to a record, and are probably not reading the actual record from disk (or for databases that live entirely in memory: into CPU cache). This gives an unfair view of the performance compared to databases that do read the data into memory (or CPU cache). While it's great that the database itself doesn't read the record, lets face it, most clients will need the actual record and not just a pointer to it. That is after all the point of a database. This also explains the unreal performance for large records. They do 30 million reads of 100 kilobyte records per second. If they were actually reading the records that would mean that their disk is doing 3 terabytes per second throughput. I want that disk!!! The hard disk and SSD also have exactly the same performance, so that means that they aren't even hitting the disk at all. So yes, they are cheating.
- easytiger 14y agoThe main issue with those stats is that they are comparing sqlite3 opened liked this: status = sqlite3_open(file_name, &db_); Writing to a file, with full ACID completeness and everything been put on this disk to some in memory key value stores. Completely different thing.
- snaky 14y ago>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We also enabled SQLite's write-ahead logging >Databases are opened in asynchronous write mode. (LevelDB's sync option, TreeDB's OAUTOSYNC option, SQLite3's synchronous options are all turned off, MDB uses the MDB_NOSYNC option, and BerkeleyDB uses the DB_TXN_WRITE_NOSYNC option). I.e., every write is pushed to the operating system, but the benchmark does not wait for the write to reach the disk. By the way there is SQLite with LMDB backend - https://gitorious.org/mdb/sqlightning https://gitorious.org/mdb/sqlightning >Using tool/speedtest.tcl in the SQLite source tree, the time to insert 1000 records on my laptop SSD was 22.42 seconds using the original code, and only 1.06 seconds using MDB. Both tests were run 3 times, with results average
- easytiger 14y agoYour point eludes me? For a start that is not tuning it. It is still using synchronous IO
- snaky 14y agoDid you look at http://symas.com/mdb/microbench/db_bench_sqlite3.cc http://symas.com/mdb/microbench/db_bench_sqlite3.cc? Search for 'write_sync'
- easytiger 14y agoAhh yea i missed, apologies std::string sync_stmt = (write_sync) ? "PRAGMA synchronous = FULL" : "PRAGMA synchronous = OFF"; Still. how is it a valid compare if it is getting it to ssd/hdd and the others are writing to memory? Just open sqlite3 with :memory:
- hyc_symas 14y agoYou're still not understanding. LMDB is a disk-based database. It is not only writing to memory.
- easytiger 14y agoMDB in those stats is writing to disk? I assume its some kind of lazy write though where the memory store is eventually persisted? in that case thats fairly impressive for a lot of use cases. Though a k/v store has a lot less to do than something like sqlite3. I'll look into it (never heard of it before). Might fix fairly well with something I'm working on right now (currently implemented using sqlite3).
- hyc_symas 14y agoIn fully synchronous mode, which is LMDB's default mode, everything is persisted upon commit. This is a memory-mapped database, not a main-memory/in-memory database. It is fully persistent to disk, it just so happens that it uses mmap() instead of read()/write() and a user-level page cache. People are always confusing "memory-mapped" with "memory-only". LMDB is not a memory-only database. http://www.openldap.org/lists/openldap-devel/201111/msg00064.html http://www.openldap.org/lists/openldap-devel/201111/msg00064...
- hosay123 14y agoI'm not sure it's fair to call this benchmark unfair.. LMDB enables use cases that simply aren't possible with non-zero-copy technologies. For example imagine a database full of large objects, and an HTTP/1.0 server that supports an API returning offsets from within these objects. In a situation like this it is possible (and efficient) to say, store 500mb-sized files in LMDB and have the first and only copy of file content be done by the kernel as it faults and copies relevant portions of file data directly into a skbuff (e.g. by passing (value_pointer + offset) directly to write(), etc.). Another use case is streamy decompression. Why read or fault an entire value into memory if the reading client disconnects after consuming only the first kilobyte? In any database where copying the entire record value occurs first, designs like the above simply aren't feasible.
- jules 14y agoThese are very niche use cases. Doing an entire benchmark on that basis without making that clear is misleading. And even in that exact use case, you will not see the massive performance difference that they are showing, because even in that case the data will have to be read from disk by the kernel, whereas in their benchmark the disk isn't even touched. Sure, in a database that doesn't support this, it might read the data from disk to memory, and then write it to a buffer, and that's going to be a bit less efficient than letting the kernel do it directly. You have an extra in memory copy, but that's still far from the performance difference that they are showing between disk read vs no disk read at all. So even in this ideal scenario, they are still massively overstating their performance (which should already be clear from the fact that no disk has 3 terabytes per second throughput). Designs like that are absolutely feasible in any database. You just have to do your own chunking. Storing 500MB records in a database is a bad idea anyways, since you never know if you're going to trip over some logic in the database, your application, or the kernel, or some interaction between these, that's going to load or copy it all unnecessarily. Better to make the chunking explicit.
- hosay123 14y agoI believe the test runs with 100 byte values, and the key is stored on the same page as the value. Therefore in order to perform a lookup the entire page must still be faulted in, and the key must still be compared, so the benchmark is not fake in that regard. Data is getting pulled off disk, just not in a way that requires multiple copies. Note the library's tiny code size (and zero copy design) has another benefit, in that it can greatly reduce cache pollution, which may again help account for the speed difference. > Storing 500MB records in a database is a bad idea anyways, since you never know if you're going to trip over some logic in the database, your application, or the kernel, or some interaction between these, that's going to load or copy it all unnecessarily. Better to make the chunking explicit. So basically you're saying there is no benefit because you aren't used to systems that support it.
- snaky 14y ago>But that probably means Why not check it yourself? http://symas.com/mdb/microbench/db_bench_mdb.cc http://symas.com/mdb/microbench/db_bench_mdb.cc Other tests source code published too, right there.
- jules 14y agoI have checked but it's impossible to tell from that, since you don't know what the API calls are doing. If somebody wants to dig into the source, that's great. However the other evidence I mentioned is enough to be quite certain of this claim, for example, that their tests do not show any difference between SSD and HDD, and that their disks would have 3 terabytes per second throughput. I know around here is it more appreciated if you state things with much hubris, even if you are only 50% sure. I worded things with a little bit of caution, even if it's 95% certain.
- hyc_symas 14y agoNonsense. Whatever the calling app does with the data will always be an additional cost over what the DB does with the data. Eliminating the DB's cost doesn't invalidate the measurement. It only shows how much waste is going on in other DB implementations. There are plenty of results in real-world apps too, not just microbenches. E.g. YCSB, Yahoo Cloud Storage Benchmark with Mapkeeper driver https://groups.google.com/forum/?fromgroups#!topic/mapkeeper-users/pPQl50fgXu4 https://groups.google.com/forum/?fromgroups#!topic/mapkeeper... MemcacheDB https://groups.google.com/forum/?fromgroups=#!topic/memcached/dxU8iO27ce4 https://groups.google.com/forum/?fromgroups=#!topic/memcache... These tests actually read data and send to a client. And of course the slapd test results, as documented in the multiple papers/presentations. http://symas.com/mdb/ http://symas.com/mdb/
- jules 14y ago> Nonsense. Whatever the calling app does with the data will always be an additional cost over what the DB does with the data. Way to ignore everything I said. What you said here is obviously not the case, as I explained, since for a large part, this is simply deferring work till later. Also, explain to me this: How is the 100kb value benchmark performing 2x as many operations per second as the 100 bytes value benchmark? Do you claim this is indicative of real world performance? And for example section 7, which does tests on the SSD. Do you really think that 30 million operations per second for 100kb values is in any relation to real world performance? Obviously, these benchmarks are not indicative of real world performance. This database may well be very fast, but these benchmarks don't show it.
- hyc_symas 14y agoWay to ignore what's printed in front of you. As the writeup clearly states, the benchmark shows that throughput is based on the number of keys in the DB, not on the size of the values. The reason the 100KB test is faster is because there are fewer keys.
- jules 14y agoSure, I get that. My question was: do you consider that indicative of real world performance? I consider that misleading. Especially if you are labeling benchmarks with tmpfs, HDD and SSD, when the read benchmarks are not even touching the disk.
- ldng 14y ago"MDB write performance when using its writable mmap option." Not really comparing apples, is it ? It doesn't seem fair to me.
- jules 14y agoWhy is that unfair?
- ldng 14y agoI did not delve into the details as you might have, but whenever I see mmap associated with huge perf I tend to wonder about corruption. I've seen to many bench comparing ondisk stores with in mmap stores flushed to disk one in a while. Then you need extra steps/configs to get good enough reliability that hinder perf.
- snaky 14y ago>whenever I see mmap associated with huge perf I tend to wonder about corruption If that was the case with lmdb implementation, that works as OpenLDAP backend all over the world, most probably you would have heard about it from the news.
- ldng 14y agoYou've have a point. But then, it's a bench mark and it would not be the first time that a bench is done using different settings than in production. Like I said, that lmdb is used in OpenLDAP doesn't automatically mean it is used with the exact same configuration as in the benchmark ;-) Nor did I say a memory store is bad, it's just you're not comparing the same things. Redis could be added to the bench for instance.
- hyc_symas 14y agoIMO main-memory stores are bad, for multiple reasons. LMDB is not a main memory store, it is a fully ACID-compliant transactional store.