3 ms·

Hi, I have some questions about Sanakirja, if you don't mind.. It stores 4kb blobs, right? Does Pijul first parse the data (copying it to other allocations), o

Hi, I have some questions about Sanakirja, if you don't mind.. It stores 4kb blobs, right? Does Pijul first parse the data (copying it to other allocations), or does it use the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, without a deserialization step (or rather, deserialization is simply a quick validation to check the data isn't corrupted), I was wondering if Pijul did anything like that. And like, is this btree header [2] stored exactly like this on disk, and does Sanakirja exploits this to avoid further copying data? (I guess there's a trouble with compression there: to decompress you really need to write in another buffer) Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer? I want to build something like Sanakirja, but with those features to avoid copying data, so I'm wondering if there's any overlap. [0] https://github.com/capnproto/capnproto-rust https://github.com/capnproto/capnproto-rust [1] https://github.com/rkyv/rkyv https://github.com/rkyv/rkyv [2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/page/index.html https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...

pmeunier 5y ago
Sanakirja does all those things indeed: Sanakirja doesn't parse anything, doesn't call read(). The answer to all your other questions is yes.
- nextaccountic 5y ago
  Thanks! It looks pretty impressive. I just saw it uses mmap. My other concern was that mmap handles I/O errors asynchronously through unix signals, and it isn't possible to pinpoint exactly which operation caused an error. Does this mean that a write error could cause data loss (other than what was written)? Or does Sanakirja guards against that? The worst case scenario is that if the program doesn't know about a write error right away, ensuing read operations on the same location could handle corrupted data, which could lead to further erroneous operations, until the error is dispatched to the signal handler and hopefully the program exits.
  - pmeunier 5y ago
    Sanakirja has atomic commits, the only hypothesis is that just a few bytes can be written to disk atomically (I don't remember, but certainly at most 8 bytes). All writes are done in atomic transactions.
    - garethrowlands 5y ago
      How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ?
    - pmeunier 5y ago
      > How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ? I don't, the OS can flush anything it wants at any time.

pmeunier 5y ago

Sanakirja does all those things indeed: Sanakirja doesn't parse anything, doesn't call read(). The answer to all your other questions is yes.

nextaccountic 5y ago
Thanks! It looks pretty impressive. I just saw it uses mmap. My other concern was that mmap handles I/O errors asynchronously through unix signals, and it isn't possible to pinpoint exactly which operation caused an error. Does this mean that a write error could cause data loss (other than what was written)? Or does Sanakirja guards against that? The worst case scenario is that if the program doesn't know about a write error right away, ensuing read operations on the same location could handle corrupted data, which could lead to further erroneous operations, until the error is dispatched to the signal handler and hopefully the program exits.
- pmeunier 5y ago
  Sanakirja has atomic commits, the only hypothesis is that just a few bytes can be written to disk atomically (I don't remember, but certainly at most 8 bytes). All writes are done in atomic transactions.
  - garethrowlands 5y ago
    How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ?
  - pmeunier 5y ago
    > How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ? I don't, the OS can flush anything it wants at any time.