5 ms·

Lightning Memory-Mapped Database Manager (LMDB) 1.0

82 points by radiator 3mo ago

radiator 3mo ago
New features in LMDB 1.0 include: - support for incremental backup - support for page-level checksums and encryption - support for DB on raw block devices - support for 2-phase commit - support for page sizes up to 64KB plus other minor additions to the API.
- Kerollmops 3mo ago
  Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel. We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words. Being able to do it in parallel is necessary, especially when you have billions of entries to operate on. [1]: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch
paveworld 3mo ago
HTTP ?? Com’on man
- radiator 3mo ago
  it is just a link to documentation
  - Retr0id 3mo ago
    TLS certs are freeeeee
    - radiator 3mo ago
      Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
      - deleted 3mo ago
        [deleted]
      - Retr0id 3mo ago
        Cryptography engineers are not excluded from being lazy sysadmins.
        radiator 3mo ago
        What do you mean "lazy"? I thought you said TLS certs were free. Do you mean they cost something after all? Time, for example? Anyway, of course in case you feel the website is a risk, you should refrain from using it. Safety comes first.
        hyc_symas 3mo ago
        Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
  - jayct 3mo ago
    that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.
    - hyc_symas 3mo ago
      If you're downloading binaries from a plaintext documentation site, I think that's on you.
  - ccapitalK 3mo ago
    Obligatory https://doesmysiteneedhttps.com/ https://doesmysiteneedhttps.com/
hmry 3mo ago
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.") But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.
- radiator 3mo ago
  It has been used successfully as the backend for OpenLDAP and Monero, at least.
- thombles 3mo ago
  Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
  - zbentley 3mo ago
    Wow, really? Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.
    - tynorf 3mo ago
      IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync. However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
hilariously 3mo ago
Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database." I generally do think read-write mode would offer higher write performance than read only as well :)
- wmanley 3mo ago
  The context is in the sentence before your quote: > The memory map can be used as a read-only or read-write map. So presumably lmdb writes to the database using the `pwrite` syscall by default, but can optionally write via the mmap instead - if you are willing to accept the increased risk of accidental data corruption.
quotemstr 3mo ago
I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access. By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back. BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS? Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.
- teravor 3mo ago
  with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.
  - quotemstr 3mo ago
    You're right about that. Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them. Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so. (Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)
initramfs 3mo ago
https://www.meilisearch.com/docs/resources/internals/storage https://www.meilisearch.com/docs/resources/internals/storage
jnwatson 3mo ago
Bummer. I'm the maintainer of the Python bindings. I have to figure out how to support both versions now...
- Kerollmops 3mo ago
  Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch). But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9... [1]: https://github.com/meilisearch/heed https://github.com/meilisearch/heed
  - justinclift 3mo ago
    heed09 and heed10 ? :)
heliskyr2 3mo ago
[flagged]
BartjeD 3mo ago
Like 8 years ago I encountered this package, I recall reading that it was designed by someone who programmed components for the Space Shuttles. I was very impressed.
erichocean 3mo ago
Another interesting LMDB fork: https://github.com/datalevin/dlmdb https://github.com/datalevin/dlmdb

82 points by radiator 3mo ago

New features in LMDB 1.0 include: - support for incremental backup - support for page-level checksums and encryption - support for DB on raw block devices - support for 2-phase commit - support for page sizes up to 64KB plus other minor additions to the API.

Kerollmops 3mo ago
Yeah, and I also added support for parallel read your own writes where you can write tons of entries and spawn multi children read-only transactions from your writes where transaction and read from them in parallel. We use this in Meilisearch [1] to post-process cache for our most common prefixes i.e., "w" will match "work", "word"... and computing this requires doing large unions of the documents matching those words. Being able to do it in parallel is necessary, especially when you have billions of entries to operate on. [1]: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch

paveworld 3mo ago

HTTP ?? Com’on man

radiator 3mo ago
it is just a link to documentation
- Retr0id 3mo ago
  TLS certs are freeeeee
  - radiator 3mo ago
    Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
    - deleted 3mo ago
      [deleted]
    - Retr0id 3mo ago
      Cryptography engineers are not excluded from being lazy sysadmins.
      - radiator 3mo ago
        What do you mean "lazy"? I thought you said TLS certs were free. Do you mean they cost something after all? Time, for example? Anyway, of course in case you feel the website is a risk, you should refrain from using it. Safety comes first.
        hyc_symas 3mo ago
        Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
- jayct 3mo ago
  that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.
  - hyc_symas 3mo ago
    If you're downloading binaries from a plaintext documentation site, I think that's on you.
- ccapitalK 3mo ago
  Obligatory https://doesmysiteneedhttps.com/ https://doesmysiteneedhttps.com/

hmry 3mo ago

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.") But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.

radiator 3mo ago
It has been used successfully as the backend for OpenLDAP and Monero, at least.
thombles 3mo ago
Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.
- zbentley 3mo ago
  Wow, really? Then what’s the point of memory mapping in the fist place? Or do they suggest manual flush/sync actions for persistence.
  - tynorf 3mo ago
    IIRC: it is to leverage the OS page cache rather than having a separate buffer pool in user land. By default lmdb uses normal pwrite/fsync for the write path, but can optionally use a writable mapping and (presumably) msync. However, some people think there are problems with this usage: (pdf warning) https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

hilariously 3mo ago

Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database." I generally do think read-write mode would offer higher write performance than read only as well :)

wmanley 3mo ago
The context is in the sentence before your quote: > The memory map can be used as a read-only or read-write map. So presumably lmdb writes to the database using the `pwrite` syscall by default, but can optionally write via the mmap instead - if you are willing to accept the increased risk of accidental data corruption.

quotemstr 3mo ago

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access. By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back. BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS? Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program. And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.

teravor 3mo ago
with mmap you also don't have to worry about committing too much system memory, if another application needs it it will start evicting your cache.
- quotemstr 3mo ago
  You're right about that. Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them. Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so. (Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)

initramfs 3mo ago

https://www.meilisearch.com/docs/resources/internals/storage https://www.meilisearch.com/docs/resources/internals/storage

jnwatson 3mo ago

Bummer. I'm the maintainer of the Python bindings. I have to figure out how to support both versions now...

Kerollmops 3mo ago
Yup! Same, I'm the maintainer of the main LMDB Rust wrapper [1] and I was maintaining heed and heed3 (because 1.0 was available from the mdb.master3 branch). But now that it's LMDB 1.0, I need to find a better way to make it be the official one but I can't really rename heed3 into heed and heed into heed-0.9... [1]: https://github.com/meilisearch/heed https://github.com/meilisearch/heed
- justinclift 3mo ago
  heed09 and heed10 ? :)

heliskyr2 3mo ago

[flagged]

BartjeD 3mo ago

Like 8 years ago I encountered this package, I recall reading that it was designed by someone who programmed components for the Space Shuttles. I was very impressed.

erichocean 3mo ago

Another interesting LMDB fork: https://github.com/datalevin/dlmdb https://github.com/datalevin/dlmdb