4 ms·

The filesystem just needs a concept of a file start offset. The initial value is 0, if you wanted to support truncating at the head, you just add to this offse

The filesystem just needs a concept of a file start offset. The initial value is 0, if you wanted to support truncating at the head, you just add to this offset the amount truncated. In a sector-based filesystem, whenever this offset goes far enough into the file that crosses sector boundaries, you reclaim those sectors as free space. When it lands within a sector somewhere, that sector is pinned and you waste some space. The problem is more that the userspace APIs don't expose well-supported mechanisms for doing this. Implementing it at the filesystem level is trivial.

cjhanks 7y ago
I can think of a cool case for this. What if you have an append-only system which is making remote backups. One service is writing the activity log append only, a second service is reading that, check-pointing, and then truncating the head of the file when the check-point has been committed. No need to do the tricky file-swap trickery.
- firethief 7y ago
  Journaling filesystems do that internally.
- dmitrygr 7y ago
  Curiously enough. PalmOS's file streams supported this very feature: delete as i read. As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end.
  - userbinator 7y ago
    As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end. Also known as a named pipe.
    - cjhanks 7y ago
      Kind of, a pipe has a limit of 16 pages which is usually 64kb.
    - dmitrygr 7y ago
      But with no limit on buffer size. Pipes have a limit after which writer blocks. Also pipes aren't persistent across reboots. These are.
- minaguib 7y ago
  Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper. See ( https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c343fa https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c... ) for some test notes I took a couple of years ago.
  - gmueckl 7y ago
    fallocate() is not supported on network file systems, is it?
nocturnial 7y ago
I thought his point in the blog was that it was indeed trivial to implement but when you think about all the consequences it becomes trickier to do it well. He makes that point at the end.
msbarnett 7y ago
This is discussed in TFA and a number of problems with this "just do" solution are pointed out.

cjhanks 7y ago

I can think of a cool case for this. What if you have an append-only system which is making remote backups. One service is writing the activity log append only, a second service is reading that, check-pointing, and then truncating the head of the file when the check-point has been committed. No need to do the tricky file-swap trickery.

firethief 7y ago
Journaling filesystems do that internally.
dmitrygr 7y ago
Curiously enough. PalmOS's file streams supported this very feature: delete as i read. As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end.
- userbinator 7y ago
  As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end. Also known as a named pipe.
  - cjhanks 7y ago
    Kind of, a pipe has a limit of 16 pages which is usually 64kb.
  - dmitrygr 7y ago
    But with no limit on buffer size. Pipes have a limit after which writer blocks. Also pipes aren't persistent across reboots. These are.
minaguib 7y ago
Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper. See ( https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c343fa https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c... ) for some test notes I took a couple of years ago.
- gmueckl 7y ago
  fallocate() is not supported on network file systems, is it?

nocturnial 7y ago

I thought his point in the blog was that it was indeed trivial to implement but when you think about all the consequences it becomes trickier to do it well. He makes that point at the end.

msbarnett 7y ago

This is discussed in TFA and a number of problems with this "just do" solution are pointed out.