3 ms·

It is actually better documented, the design is explained in the code: https://docs.rs/sanakirja-core https://docs.rs/sanakirja-core See for instance https://d

It is actually better documented, the design is explained in the code: https://docs.rs/sanakirja-core https://docs.rs/sanakirja-core See for instance https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/page/index.html https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p... Sanakirja even reached a point where bugs were found only in the last few functions without comments. Writing the comments to explain exactly what each line does in all cases (there are lots of cases everywhere) fixed everything.

nextaccountic 5y ago
Hi, I have some questions about Sanakirja, if you don't mind.. It stores 4kb blobs, right? Does Pijul first parse the data (copying it to other allocations), or does it use the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, without a deserialization step (or rather, deserialization is simply a quick validation to check the data isn't corrupted), I was wondering if Pijul did anything like that. And like, is this btree header [2] stored exactly like this on disk, and does Sanakirja exploits this to avoid further copying data? (I guess there's a trouble with compression there: to decompress you really need to write in another buffer) Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer? I want to build something like Sanakirja, but with those features to avoid copying data, so I'm wondering if there's any overlap. [0] https://github.com/capnproto/capnproto-rust https://github.com/capnproto/capnproto-rust [1] https://github.com/rkyv/rkyv https://github.com/rkyv/rkyv [2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/page/index.html https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
- pmeunier 5y ago
  Sanakirja does all those things indeed: Sanakirja doesn't parse anything, doesn't call read(). The answer to all your other questions is yes.
  - nextaccountic 5y ago
    Thanks! It looks pretty impressive. I just saw it uses mmap. My other concern was that mmap handles I/O errors asynchronously through unix signals, and it isn't possible to pinpoint exactly which operation caused an error. Does this mean that a write error could cause data loss (other than what was written)? Or does Sanakirja guards against that? The worst case scenario is that if the program doesn't know about a write error right away, ensuing read operations on the same location could handle corrupted data, which could lead to further erroneous operations, until the error is dispatched to the signal handler and hopefully the program exits.
    - pmeunier 5y ago
      Sanakirja has atomic commits, the only hypothesis is that just a few bytes can be written to disk atomically (I don't remember, but certainly at most 8 bytes). All writes are done in atomic transactions.
      - garethrowlands 5y ago
        How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ?
      - pmeunier 5y ago
        > How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ? I don't, the OS can flush anything it wants at any time.
samatman 5y ago
Congrats on the beta release! I've been following along for awhile with great interest, looking forward to picking out a couple repos and shadowing git with pijul now that it's beta. Since you mentioned sanakirja, I'd like to ask: the obvious (to me) reason to build it, rather than use SQLite, is easy ABI compatibility with the rest of the project through Rust. Are there other architectural reasons it's particularly suited to pijul? I ask because I already store as much as I can in SQLite databases, and am moving to patchsets and changesets for certain kinds of document-oriented data management. Storing pijul's artifacts in that same system would be a natural fit for my application, so I wonder how closely tied together the two codebases are on a practical level.
- pmeunier 5y ago
  > Are there other architectural reasons it's particularly suited to pijul? Yes, I originally wrote it because I wanted to fork tables without copying any byte. This is how channels (similar to Git branches) are implemented in Pijul. > I wonder how closely tied together the two codebases are on a practical level. Pijul can use a generic backend, although in practice only variants of Sanakirja have been implemented. Sanakirja is generic enough to handle different workflows, although large values aren't implemented yet (they aren't hard to implement, I just never needed them and nobody else did it).
  - samatman 5y ago
    Generic backend is wonderful news! As it happens SQLite added some of the machinery from fossil for no-copy tables and changesets as an optional extension. I mention this because it gives me hope that I might cobble something efficient together in SQLite, I could offer a few reasons I'm glad Sanakirja exists aside from pijul and the advantage to your project of being able to tweak both sides of the system is compelling. Congratulations again on the beta. Doing hard and basic work for years is to be commended.

nextaccountic 5y ago

Hi, I have some questions about Sanakirja, if you don't mind.. It stores 4kb blobs, right? Does Pijul first parse the data (copying it to other allocations), or does it use the data as is? I mean, there are some libraries like cap'n'proto[0] and rkyv[1] that can directly use the file contents as an in-memory data structure, without a deserialization step (or rather, deserialization is simply a quick validation to check the data isn't corrupted), I was wondering if Pijul did anything like that. And like, is this btree header [2] stored exactly like this on disk, and does Sanakirja exploits this to avoid further copying data? (I guess there's a trouble with compression there: to decompress you really need to write in another buffer) Also, is the I/O done with something that prevent userspace copies like mmap or io_uring, or does it eventually calls read() to copy the data to its own buffer? I want to build something like Sanakirja, but with those features to avoid copying data, so I'm wondering if there's any overlap. [0] https://github.com/capnproto/capnproto-rust https://github.com/capnproto/capnproto-rust [1] https://github.com/rkyv/rkyv https://github.com/rkyv/rkyv [2] https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/page/index.html https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...

pmeunier 5y ago
Sanakirja does all those things indeed: Sanakirja doesn't parse anything, doesn't call read(). The answer to all your other questions is yes.
- nextaccountic 5y ago
  Thanks! It looks pretty impressive. I just saw it uses mmap. My other concern was that mmap handles I/O errors asynchronously through unix signals, and it isn't possible to pinpoint exactly which operation caused an error. Does this mean that a write error could cause data loss (other than what was written)? Or does Sanakirja guards against that? The worst case scenario is that if the program doesn't know about a write error right away, ensuing read operations on the same location could handle corrupted data, which could lead to further erroneous operations, until the error is dispatched to the signal handler and hopefully the program exits.
  - pmeunier 5y ago
    Sanakirja has atomic commits, the only hypothesis is that just a few bytes can be written to disk atomically (I don't remember, but certainly at most 8 bytes). All writes are done in atomic transactions.
    - garethrowlands 5y ago
      How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ?
    - pmeunier 5y ago
      > How do you avoid the OS randomly deciding to write a page to disk, per https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/ ? I don't, the OS can flush anything it wants at any time.

samatman 5y ago

Congrats on the beta release! I've been following along for awhile with great interest, looking forward to picking out a couple repos and shadowing git with pijul now that it's beta. Since you mentioned sanakirja, I'd like to ask: the obvious (to me) reason to build it, rather than use SQLite, is easy ABI compatibility with the rest of the project through Rust. Are there other architectural reasons it's particularly suited to pijul? I ask because I already store as much as I can in SQLite databases, and am moving to patchsets and changesets for certain kinds of document-oriented data management. Storing pijul's artifacts in that same system would be a natural fit for my application, so I wonder how closely tied together the two codebases are on a practical level.

pmeunier 5y ago
> Are there other architectural reasons it's particularly suited to pijul? Yes, I originally wrote it because I wanted to fork tables without copying any byte. This is how channels (similar to Git branches) are implemented in Pijul. > I wonder how closely tied together the two codebases are on a practical level. Pijul can use a generic backend, although in practice only variants of Sanakirja have been implemented. Sanakirja is generic enough to handle different workflows, although large values aren't implemented yet (they aren't hard to implement, I just never needed them and nobody else did it).
- samatman 5y ago
  Generic backend is wonderful news! As it happens SQLite added some of the machinery from fossil for no-copy tables and changesets as an optional extension. I mention this because it gives me hope that I might cobble something efficient together in SQLite, I could offer a few reasons I'm glad Sanakirja exists aside from pijul and the advantage to your project of being able to tweak both sides of the system is compelling. Congratulations again on the beta. Doing hard and basic work for years is to be commended.