6 ms·
I second the recomendation of LMDB. With one important caveat: under heavy write load it is perfect demonstration of brokenness of semaphore implementation on f
by dfox 6y ago
I second the recomendation of LMDB. With one important caveat: under heavy write load it is perfect demonstration of brokenness of semaphore implementation on freebsd and macos.
- skrebbel 6y agoIn what way is LMDB better than eg SQLite or Redis? For what kinds of use cases would you recommend it?
- corysama 6y agoIt's very simple. Single C file implementation. Binary blob keys : binary blob values. The end. If all you need is a bunch of blobs written and read back reliably while in a statistically unreliable situation, LMDB is great.
- namibj 6y agoWell, sqlite is also just a single .c with a single .h.
- tlb 6y agoSqlite is dozens of files. See https://www3.sqlite.org/cgi/src/dir?ci=tip&name=src https://www3.sqlite.org/cgi/src/dir?ci=tip&name=src
- samatman 6y agoYou're both right... kinda. SQLite is dozens of files, if you're browsing it, or modifying it. Which you can do, as long as you're ok with contributing your changes being impossible, since it's open-source but closed-contribution. If you're compiling it, it is, in fact, one .c and one .h: https://www.sqlite.org/amalgamation.html https://www.sqlite.org/amalgamation.html If you're choosing between LMDB and SQLite, the latter sense is the relevant one, so they're identical along this particular axis.
- dfox 6y agoFor my applications: latency. The fact that writers do not block readers is just a nice bonus.
- kortex 6y agoDo you reckon LMDB would be reasonably performant compared to raw mmaped files for zero-copy passing large images between processes? I want something like a ring buffer but mitigate the risk of use-after-free if the consumer lags. Seems like lmdb automatically re-uses memory that's long been freed but is sensible to detect that an incoming write needs more space. Python/c++. I've looked into mmap and flock but it's messy and not highly portable.
- dfox 6y agoLMDB is in fact not much more than large mmaped file and clever synchronization mechanism on top of that that also serves as dictionary implementation and transaction mechanism. My uses of LMDB are essentially replacements for mmaped file and LMDB gives me the ability to sanely do partial updates.
- kortex 6y agoSounds exactly like what I'm looking for, definitely will be digging into it more. Thanks!
- hyc_symas 6y agoLMDB and SQLite are not directly comparable. LMDB is a transactional B+tree-based key/value store. SQLite is an implementation of a transactional SQL data model on top of a B+tree-based key/value store, so it is logically at least one abstraction layer higher than LMDB. (Key/value stores underlie pretty much all of the other data models you'll ever use.) That aside - LMDB is not just smaller, faster, and more reliable than SQLite, it is also smaller/faster/more reliable than SQLite's own B+tree implementation, and SQLite can be patched to use LMDB instead of its own B+tree code, resulting in a smaller/faster footprint for SQLite itself. Proof of concept was done here https://github.com/LMDB/sqlightning https://github.com/LMDB/sqlightning A new team has picked this up and carried it forward https://github.com/LumoSQL/LumoSQL https://github.com/LumoSQL/LumoSQL Generally, unless your application has fairly simple data storage needs, it's better to use some other data model built on top of LMDB than to use it (or any K/V store) directly. (But if building data storage servers and implementing higher level data models is your thing, then you'd most likely be building directly on top of LMDB.)