2 ms·
> Sure you can. Where's the source code for that? (I read the paper when it came out, good stuff!) In my experience, I don't fully trust research results—espe
by erichocean 10y ago
> Sure you can.
Where's the source code for that? (I read the paper when it came out, good stuff!)
In my experience, I don't fully trust research results—especially for databases—until they've been replicated in production. Especially on high-performance systems, small changes (oh, we added error handling, or oh, now we're actually calling fsync) can easily drop actual achieved performance 1 or even 2 orders of magnitude. (I'm not saying this is true for the linked paper, just that I'd want to verify the actual achieved performance on my own workloads before I switched to it. I do believe it's possible FWIW, I just haven't seen it in the real world.)
So if there's an LMDB-style codebase out there I can spin up that uses a Bw-tree internally, I'm definitely interested and would love a link. :-)
Also FWIW, what I like about LMDB isn't just the speed, it's the entire architecture. This fall I'm looking into using ScyllaDB for page storage, and storing the meta pages inside CockroachDB, with a user-per-LMDB-environment model. LMDB's essential design makes that (relatively) easy, whereas with, say, RocksDB, not so much (at least for me).
- jasonwatkinspdx 10y ago> Where's the source code for that? Closed. It's backing Hekaton in MS SQL and some features of Azure DocumentDB. While there are many systems in the literature that demonstrate some variation of lock free log and indirection map, they're all either research demonstration codebases or closed source to the best of my knowledge. The reason I have an axe to grind here is I think the generalizations around LMDB made by you and others are making this situation worse, by convincing people that haven't read the last few years of literature that LMDB's design is all there is. LMDB is great but narrow. It has no answer for many workloads. It's inevitable that better things will be devised. I want those things to be FOSS. Do the over-reaching and over-simplifying comments about LMDB make that more or less likely?
- erichocean 10y ago> I think the generalizations around LMDB made by you and others are making this situation worse Fair enough. For anyone else following, @jasonwatkinspdx is correct that LMDB is not the right (= best available today) choice for certain workloads. I personally used TrailDB for event traces and if I needed prefix compression on keys (like the CockroachDB team), RocksDB would be an arguably better choice. What I don't agree with is that LMDB's existence or users are causing open source database research to stagnate. Quite the contrary, I think the benefits of LMDB's architecture have not yet been fully realized and I would love to see more research be done to improve/extend it. Most of the research action today (at least to me) seems focused on making log-structure merge trees and variations thereof less sucky at the margin instead of identifying what makes LMDB's design actually tick and then improving on it. There's still a lot of low-hanging fruit in the LMDB architecture waiting to be explored.
- jasonwatkinspdx 10y agoThanks for a really civil reply to what was admittedly a pretty barbed comment on my part. I wasn't aware of TrailDB, I'll check it out. Thanks!