4 ms·

> Because fsync() is so coarse You can do some work in advance with sync_file_range. fsync is still needed to write out the metadata, but it'll have to do less

> Because fsync() is so coarse You can do some work in advance with sync_file_range. fsync is still needed to write out the metadata, but it'll have to do less work and you may be able to get better pacing and backpressure that way. > Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them. Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Another issue might be that multiple transactions could overlap and be of unbounded size, so now you would need MVCC, exclusive locking or the ability for the kernel to provide feedback to userspace that it will immediately rollback the transaction due to resource limits or conflicts. This suddenly looks more and more like a database than a posix filesystem. There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Lazy writes that the kernel can flush to disk whenever it feels like provide a lot of the performance that we take for granted when working with large amounts of data. > And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.

BlueTemplar 6y ago
> CoW Are you talking about copy-on-write or redirect-on-write, which seems to be the real copy on write ? https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write
- the8472 6y ago
  From the same article: > Despite their names, copy-on-write usually refers to the first technique.
  - BlueTemplar 6y ago
    Yes, hence my question.
    - the8472 6y ago
      CoW filesystem is a standing term used to refer to things such as zfs or btrfs. They use the first kind.
josephg 6y ago
> Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Thats fine. Legacy filesystems can just use a best-effort approach like they do now with 95% of programs. I already wouldn't put data I care about on ext2. Programs could detect transaction support with a syscall then decide whether to fallback to old behaviour, lose reliability or for a database maybe just refuse to run entirely. > This suddenly looks more and more like a database than a posix filesystem. POSIX filesystem apis aren't fit for use when you care about correctness. And as I understand it ZFS, btrfs and APFS already look more like databases internally than traditional posix filesystems. If that ship is sailing anyway we may as well get all the benefits we can. > There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Are there? I can't think of many. Maybe copying big files? We'll need to keep supporting the current unsafe-by-default posix API forever for current software anyway. I just don't think it should be the default for 95% of applications. When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes. If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. It seems like a pity though, and I think the filesystem is worth salvaging. And with the API I'm suggesting, I don't think large operations need to get much slower. The transactional API could just optimistically write at full speed to unlinked sectors on disk, then atomically update pointers to reference the new sectors when the transaction is committed. That wouldn't add much amortized overhead. I'm sure there's more complexity than that in practice, but it sounds pretty tractable and fast in the common case of a single writer. (And in the slow case where multiple programs try to fight over the same file, slow would be an improvement over the current behaviour of "probably everything breaks".) > I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace. Its still almost impossible to write a database correctly on top of the POSIX APIs. Especially if you want it to be correct. That seems fixable. We should fix it.
- the8472 6y ago
  > If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. I don't mean everyone should use sqlite. We just need more safe abstractions over posix APIs that cover different use-cases. If you need to concurrently query and modify many small records, then sure, go with sqlite. > Are there? I can't think of many. Maybe copying big files? It's not just the big files. Millions of small files would cause some overhead if you actually tried commiting them instead of relying on best-effort behavior (see git case below). Temp and cache directories. Archival. Compiler artifacts. Often you don't even want them to actually hit the disk at all if you have enough RAM. A file can act as nothing more than named swap space. > When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes. Sure, that's for the canonical git storage, which rarely touches many files anyway. But if you checkout a worktree (e.g. by switching branches) then putting all the thousands of files from the .git object storage into the work tree does not need durability. You want to be able to work on those files as soon as possible, even if they're just in the page cache and not written to disk yet. The command shouldn't block on fsync.
phkahler 6y ago
>> Many existing filesystems simply cannot support such an API because they're not journaling or CoW. So let's get rid of them. And that leads to "but what about all the software that hasn't updated to the new APIs?" Guess what?

BlueTemplar 6y ago

> CoW Are you talking about copy-on-write or redirect-on-write, which seems to be the real copy on write ? https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write

the8472 6y ago
From the same article: > Despite their names, copy-on-write usually refers to the first technique.
- BlueTemplar 6y ago
  Yes, hence my question.
  - the8472 6y ago
    CoW filesystem is a standing term used to refer to things such as zfs or btrfs. They use the first kind.

josephg 6y ago

> Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Thats fine. Legacy filesystems can just use a best-effort approach like they do now with 95% of programs. I already wouldn't put data I care about on ext2. Programs could detect transaction support with a syscall then decide whether to fallback to old behaviour, lose reliability or for a database maybe just refuse to run entirely. > This suddenly looks more and more like a database than a posix filesystem. POSIX filesystem apis aren't fit for use when you care about correctness. And as I understand it ZFS, btrfs and APFS already look more like databases internally than traditional posix filesystems. If that ship is sailing anyway we may as well get all the benefits we can. > There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Are there? I can't think of many. Maybe copying big files? We'll need to keep supporting the current unsafe-by-default posix API forever for current software anyway. I just don't think it should be the default for 95% of applications. When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes. If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. It seems like a pity though, and I think the filesystem is worth salvaging. And with the API I'm suggesting, I don't think large operations need to get much slower. The transactional API could just optimistically write at full speed to unlinked sectors on disk, then atomically update pointers to reference the new sectors when the transaction is committed. That wouldn't add much amortized overhead. I'm sure there's more complexity than that in practice, but it sounds pretty tractable and fast in the common case of a single writer. (And in the slow case where multiple programs try to fight over the same file, slow would be an improvement over the current behaviour of "probably everything breaks".) > I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace. Its still almost impossible to write a database correctly on top of the POSIX APIs. Especially if you want it to be correct. That seems fixable. We should fix it.

the8472 6y ago
> If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. I don't mean everyone should use sqlite. We just need more safe abstractions over posix APIs that cover different use-cases. If you need to concurrently query and modify many small records, then sure, go with sqlite. > Are there? I can't think of many. Maybe copying big files? It's not just the big files. Millions of small files would cause some overhead if you actually tried commiting them instead of relying on best-effort behavior (see git case below). Temp and cache directories. Archival. Compiler artifacts. Often you don't even want them to actually hit the disk at all if you have enough RAM. A file can act as nothing more than named swap space. > When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes. Sure, that's for the canonical git storage, which rarely touches many files anyway. But if you checkout a worktree (e.g. by switching branches) then putting all the thousands of files from the .git object storage into the work tree does not need durability. You want to be able to work on those files as soon as possible, even if they're just in the page cache and not written to disk yet. The command shouldn't block on fsync.

phkahler 6y ago

>> Many existing filesystems simply cannot support such an API because they're not journaling or CoW. So let's get rid of them. And that leads to "but what about all the software that hasn't updated to the new APIs?" Guess what?