3 ms·

Is this a problem that really needs a solution? I can't recall any times where this would have been useful to me or think of a situation where it'd be a deal b

by bagels 7y ago

Is this a problem that really needs a solution? I can't recall any times where this would have been useful to me or think of a situation where it'd be a deal breaker. I could contrive a bunch of other operations that are inefficient on file systems like this (remove every other byte from the file, why not?), but doesn't mean that literally any idea we can think up should be supported by the file system apis.

buckminster 7y ago
One place this would be useful is a DVR. You want to record the last n hours of video and discard anything older. And it's not inefficient. It just requires a lot of careful engineering that's probably not worth the bother.
- frei 7y ago
  It's unnecessary to create a filesystem-level support I think, since you could make a circular buffer holding a video stream in a file with some constant size that is approximately n hours of video. With some clever tricks you could even keep the format compatible with existing video container formats.
  - eps 7y ago
    Or just use a set of files, each holding N minutes with, perhaps, some overlap + an index file. No real need to over-engineer things here.
taffer 7y ago
Filesystem design and operating system design is full of these kinds of questions. From Unix and Windows we are used that we can insert and delete the end of a file and that we can read, write and seek anywhere in the file, but a filesystem could potentially do much more: Inserting/deleting at the beginning or inserting/deleting in the middle. Deleting at the beginning of a file would be handy for queues or logs and inserting and deleting in the middle is a common operation for text editors. There are even more questions you could ask: - Why not have tabular files that allow indexed access? - Do we need hierarchical directories or do we need directories at all? - What about adding tags to files? - Why having a filesystem at all?
el_benhameen 7y ago
From the ones I’ve read, Raymond Chen’s articles are more about problems that are fun to think about than problems that need solutions (though there are plenty of articles about problems that fit both descriptions).
kelnos 7y ago
At a previous job, long ago, I was working on software to add tags to audio and video files. Some of the formats required modifying or adding data to the start of the file. Having to rewrite a multi-gigabyte file just to add a few hundred bytes to the beginning of the file is slow, and dangerous if there isn't room on the disk for a second copy of the file. (Not as much of a consideration now, but disks weren't as large or cheap in the mid-00s.)
chewbacha 7y ago
I think this was more for fun and intrigue than for real > And I suspect the fact that you can simulate the operation yourself is a major reason why the feature doesn’t exist: Time and effort is better-spent adding features that applications couldn’t simulate on their own.
andreareina 7y ago
I just recently had to do this at work. We load data into aws redshift from csv in s3. Well, one particular set of files had a BOM[1], which redshift chokes on. Cue having to rewrite the entire 4.5GB file because to remove the first three bytes. [1] https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark

by bagels 7y ago

buckminster 7y ago

One place this would be useful is a DVR. You want to record the last n hours of video and discard anything older. And it's not inefficient. It just requires a lot of careful engineering that's probably not worth the bother.

frei 7y ago
It's unnecessary to create a filesystem-level support I think, since you could make a circular buffer holding a video stream in a file with some constant size that is approximately n hours of video. With some clever tricks you could even keep the format compatible with existing video container formats.
- eps 7y ago
  Or just use a set of files, each holding N minutes with, perhaps, some overlap + an index file. No real need to over-engineer things here.

taffer 7y ago

Filesystem design and operating system design is full of these kinds of questions. From Unix and Windows we are used that we can insert and delete the end of a file and that we can read, write and seek anywhere in the file, but a filesystem could potentially do much more: Inserting/deleting at the beginning or inserting/deleting in the middle. Deleting at the beginning of a file would be handy for queues or logs and inserting and deleting in the middle is a common operation for text editors. There are even more questions you could ask: - Why not have tabular files that allow indexed access? - Do we need hierarchical directories or do we need directories at all? - What about adding tags to files? - Why having a filesystem at all?

el_benhameen 7y ago

From the ones I’ve read, Raymond Chen’s articles are more about problems that are fun to think about than problems that need solutions (though there are plenty of articles about problems that fit both descriptions).

kelnos 7y ago

At a previous job, long ago, I was working on software to add tags to audio and video files. Some of the formats required modifying or adding data to the start of the file. Having to rewrite a multi-gigabyte file just to add a few hundred bytes to the beginning of the file is slow, and dangerous if there isn't room on the disk for a second copy of the file. (Not as much of a consideration now, but disks weren't as large or cheap in the mid-00s.)

chewbacha 7y ago

I think this was more for fun and intrigue than for real > And I suspect the fact that you can simulate the operation yourself is a major reason why the feature doesn’t exist: Time and effort is better-spent adding features that applications couldn’t simulate on their own.

andreareina 7y ago

I just recently had to do this at work. We load data into aws redshift from csv in s3. Well, one particular set of files had a BOM[1], which redshift chokes on. Cue having to rewrite the entire 4.5GB file because to remove the first three bytes. [1] https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark