2 ms·

> If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more tha

> If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. I don't mean everyone should use sqlite. We just need more safe abstractions over posix APIs that cover different use-cases. If you need to concurrently query and modify many small records, then sure, go with sqlite. > Are there? I can't think of many. Maybe copying big files? It's not just the big files. Millions of small files would cause some overhead if you actually tried commiting them instead of relying on best-effort behavior (see git case below). Temp and cache directories. Archival. Compiler artifacts. Often you don't even want them to actually hit the disk at all if you have enough RAM. A file can act as nothing more than named swap space. > When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes. Sure, that's for the canonical git storage, which rarely touches many files anyway. But if you checkout a worktree (e.g. by switching branches) then putting all the thousands of files from the .git object storage into the work tree does not need durability. You want to be able to work on those files as soon as possible, even if they're just in the page cache and not written to disk yet. The command shouldn't block on fsync.

josephg 6y ago
> We just need more safe abstractions over posix APIs that cover different use-cases. I reach for lmdb for a lot of small projects. It feels like failure though. We shouldn't need things like that just to deterministically save data to disk. > The command shouldn't block on fsync. Sure - some operations can and should discard durability for higher performance. Sounds great to me. But maybe you should opt in to that. It certainly shouldn't be the only behaviour the kernel provides. If Inkscape crashes halfway through saving my work, it shouldn't by default end up in a half-saved state where my file is permanently corrupted. But we can have our cake and eat it too here. It can be as simple as, if you don't wrap your write() calls in a filesystem transaction, the behaviour stays the same as it always was. Or maybe the default behaviour is that a write call entirely happens or entirely doesn't happen, but you can optionally ioctl(NO_DURABILITY) or something for extra performance in git checkout. I hear you that git checkout performance (and things like it) are important to you. Keeping the current performance of those operations sounds good to me too!

josephg 6y ago

> We just need more safe abstractions over posix APIs that cover different use-cases. I reach for lmdb for a lot of small projects. It feels like failure though. We shouldn't need things like that just to deterministically save data to disk. > The command shouldn't block on fsync. Sure - some operations can and should discard durability for higher performance. Sounds great to me. But maybe you should opt in to that. It certainly shouldn't be the only behaviour the kernel provides. If Inkscape crashes halfway through saving my work, it shouldn't by default end up in a half-saved state where my file is permanently corrupted. But we can have our cake and eat it too here. It can be as simple as, if you don't wrap your write() calls in a filesystem transaction, the behaviour stays the same as it always was. Or maybe the default behaviour is that a write call entirely happens or entirely doesn't happen, but you can optionally ioctl(NO_DURABILITY) or something for extra performance in git checkout. I hear you that git checkout performance (and things like it) are important to you. Keeping the current performance of those operations sounds good to me too!