5 ms·
The presentation and the paper pretty much describe how LMDB works already. https://symas.com/lightning-memory-mapped-database/ https://symas.com/lightning-mem
by nuopnu 9y ago
The presentation and the paper pretty much describe how LMDB works already.
https://symas.com/lightning-memory-mapped-database/ https://symas.com/lightning-memory-mapped-database/
Except, it allows only a single writer and it's not directly byte-addressable, unless you make a request to preallocate a chunk of fixed-size mmap'ed memory.
- valarauca1 9y agoYes but in byte-addressable mode you can trust the map is fixed, which doesn't seem to be the case here.
- nuopnu 9y agoNot exactly. I'm not a user of this, but http://www.lmdb.tech/doc/group__mdb__env.html#ga492952277c481bc4a6fa08ef71c29487 http://www.lmdb.tech/doc/group__mdb__env.html#ga492952277c48... Ctrl+f through the doc for "MDB_FIXEDMAP" for more details. So there is _some_ support, where you can expect to have the whole file at a fixed address. What is not handled is that your data may be moved within the file, and there's nothing you can do about it, short of keeping a long-lived read only transaction: http://www.lmdb.tech/doc/todo.html http://www.lmdb.tech/doc/todo.html
- hyc_symas 9y agoYou have it right. In practice, nobody has wanted support for this so it has remained unimplemented. Note that if your data items are large enough (more than 1/2 a page) they'll go into their own overflow pages, and then won't shift around at all. (But if you delete/modify a value, the new version will of course be in a new overflow page.) Granted, this could be a case of "if you build it they will come" - before LMDB existed, nobody cared about mmap'd transactional DBs either. And personally, I still see good use cases for it.
- cryptonector 9y agoNote that the LMDB approach, and especially if you want MDB_FIXEDMAP, limits DB size to the largest mmap()ing you can get at a fixed location. That's not good in a world of 48-bit address spaces.
- nuopnu 9y agoPractically it's even less than 47 bits. :) But sure, know your limits.
- cryptonector 9y agoYes, I know :)
- jarulraj 9y agoLMDB is a highly-optimized storage engine. However, it is not a full-featured DBMS (e.g., no support for query optimization). This talk focuses on the potential impact of NVM on different layers of a full-featured DBMS -- not only the storage engine.