4 ms·

Even with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enoug

by tgbugs 6y ago

Even with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enough to find, much less modify an sqlite database can make their own mess if they want to. I'm more worried about cases where something like a sidecar file gets deleted or overwritten, or fails to be copied along with the main file. Some programs and file formats come up with their own way of embedding sidecar metadata in the main file, but that is a complete hack that has to be used to work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable.

skissane 6y ago
> Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable. On Solaris, xattrs really are up to the task; they are basically Windows alternate data streams by another name. Unlike Linux xattrs, their maximum size is the same as that of the main stream of the file. (Likewise, on macOS the maximum xattr size is the same as that of a file's main stream.) I think if Linux would lift its limit on maximum xattr size then you would have the same basic function across most major operating systems, just with some differences in API. Then if someone could devise a standard API (even as just a portability layer), your problem would be mostly solved. Just also making sure that tools like cp, scp, sftp, zip, tar, etc, know about the xattrs and transport them. (Which is actually already true for some of those tools on some platforms.)
- tgbugs 6y ago
  I have used python's pathlib to create a common abstraction across ADS and linux xattrs, but have had to stop at the lowest common denominator that is the linux implementation. It is good to know that some of the other nixen don't have the same restrictions because it means there might be a shorter route to interoperability (especially in light of some of the other threads on the linux file system that have popped up over the last couple of days). The second piece of the puzzle is also exactly as you point out, which is xattr support across all the common tools. Many of them have it already, however it is usually not enabled by default, so it is extremely easy for a user to accidentally copy only the main file data. To give an example of the issue, vim's default save implementation copies files in a way that removes xattrs. Thus, even if xattrs can get closer, the fact that they still require tools to implement correct behaviour makes is a bit like the situation with cooperative multitasking.
- sneak 6y ago
  As a sysadmin, this sounds like a mess. Some tools support them, and some don't? Does the alternate stream/fork data show up in `df` and `du`? Can I cat the alternate streams? How many alternate streams are there? Why not just leave files as a sequence of bytes and let the applications handle this in their file formats? It's not like we don't have standards and libraries there already.
  - kortex 6y ago
    They already aren't a stream of bytes. There's time and permission metadata, which tar, cp, rsync and friends may or may not handle in various ways based on option flags. I guess it boils down to what is "application data" and what isn't.
- kortex 6y ago
  This infuriates me. There are so many problems I run into, day-to-day even, which could be readily solved if files/folders just had a stringly-typed key-value metadata store of arbitrary size which was portable-ish across file systems. Heck, even just one bytes field I can dump json or some serde into, that doesn't impinge on the actual file bytestream. xattr is close but no cigar. Instead you either need sidecar files, archives, or custom binary formats. It's as if the posix filesystem abstraction crystallized a bit too soon (same like tcp/ip)
  - rwallace 6y ago
    I'm curious about the specifics of the problems this would solve, having been doing some thinking about what an ideal file system would look like. In particular, is this something you would need primarily for files you create, or existing files created by other people? And, to be fully useful, would it need to allow the attaching of such metadata to arbitrary files/folders regardless of type?
enriquto 6y ago
> work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Hmmm... have you ever heard about the concept of archive? It's a single file that you can copy around, but it contains literally several files inside!
- johnsoft 6y ago
  Snark aside, this is obviously not a silver bullet. First reason that comes to mind: if you delete a file on the file system, the disk space can be reclaimed instantly. But if you delete a file in an archive, you must either rewrite the archive (since you can't "move" bytes in a file), or accept that disk space will be wasted.
  - skissane 6y ago
    > But if you delete a file in an archive, you must either rewrite the archive (since you can't "move" bytes in a file), or accept that disk space will be wasted. Another option is to tell the operating system to deallocate part of the file, turning it into a sparse file. This can be done with fallocate(FALLOC_FL_PUNCH_HOLE) on Linux, FSCTL_SET_ZERO_DATA on Windows. Linux also has fallocate(FALLOC_FL_COLLAPSE_SIZE) which lets you actually remove a byte range from the middle of a file, shifting up everything that comes after that byte range. So, in fact, you can "move" bytes in a file, on certain filesystems on Linux (ext4, xfs, among others). Unfortunately, support for sparse files is quite variable across platforms. The macOS equivalent to fallocate(FALLOC_FL_PUNCH_HOLE) if fcntl(F_PUNCHHOLE), but the issue is HFS+ has never supported sparse files. UFS did, but UFS support was dropped from 10.7 onwards. The support is supposed to be back in APFS, but I don't know if it actually works.

by tgbugs 6y ago

skissane 6y ago

> Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable. On Solaris, xattrs really are up to the task; they are basically Windows alternate data streams by another name. Unlike Linux xattrs, their maximum size is the same as that of the main stream of the file. (Likewise, on macOS the maximum xattr size is the same as that of a file's main stream.) I think if Linux would lift its limit on maximum xattr size then you would have the same basic function across most major operating systems, just with some differences in API. Then if someone could devise a standard API (even as just a portability layer), your problem would be mostly solved. Just also making sure that tools like cp, scp, sftp, zip, tar, etc, know about the xattrs and transport them. (Which is actually already true for some of those tools on some platforms.)

tgbugs 6y ago
I have used python's pathlib to create a common abstraction across ADS and linux xattrs, but have had to stop at the lowest common denominator that is the linux implementation. It is good to know that some of the other nixen don't have the same restrictions because it means there might be a shorter route to interoperability (especially in light of some of the other threads on the linux file system that have popped up over the last couple of days). The second piece of the puzzle is also exactly as you point out, which is xattr support across all the common tools. Many of them have it already, however it is usually not enabled by default, so it is extremely easy for a user to accidentally copy only the main file data. To give an example of the issue, vim's default save implementation copies files in a way that removes xattrs. Thus, even if xattrs can get closer, the fact that they still require tools to implement correct behaviour makes is a bit like the situation with cooperative multitasking.
sneak 6y ago
As a sysadmin, this sounds like a mess. Some tools support them, and some don't? Does the alternate stream/fork data show up in `df` and `du`? Can I cat the alternate streams? How many alternate streams are there? Why not just leave files as a sequence of bytes and let the applications handle this in their file formats? It's not like we don't have standards and libraries there already.
- kortex 6y ago
  They already aren't a stream of bytes. There's time and permission metadata, which tar, cp, rsync and friends may or may not handle in various ways based on option flags. I guess it boils down to what is "application data" and what isn't.
kortex 6y ago
This infuriates me. There are so many problems I run into, day-to-day even, which could be readily solved if files/folders just had a stringly-typed key-value metadata store of arbitrary size which was portable-ish across file systems. Heck, even just one bytes field I can dump json or some serde into, that doesn't impinge on the actual file bytestream. xattr is close but no cigar. Instead you either need sidecar files, archives, or custom binary formats. It's as if the posix filesystem abstraction crystallized a bit too soon (same like tcp/ip)
- rwallace 6y ago
  I'm curious about the specifics of the problems this would solve, having been doing some thinking about what an ideal file system would look like. In particular, is this something you would need primarily for files you create, or existing files created by other people? And, to be fully useful, would it need to allow the attaching of such metadata to arbitrary files/folders regardless of type?

enriquto 6y ago

> work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Hmmm... have you ever heard about the concept of archive? It's a single file that you can copy around, but it contains literally several files inside!

johnsoft 6y ago
Snark aside, this is obviously not a silver bullet. First reason that comes to mind: if you delete a file on the file system, the disk space can be reclaimed instantly. But if you delete a file in an archive, you must either rewrite the archive (since you can't "move" bytes in a file), or accept that disk space will be wasted.
- skissane 6y ago
  > But if you delete a file in an archive, you must either rewrite the archive (since you can't "move" bytes in a file), or accept that disk space will be wasted. Another option is to tell the operating system to deallocate part of the file, turning it into a sparse file. This can be done with fallocate(FALLOC_FL_PUNCH_HOLE) on Linux, FSCTL_SET_ZERO_DATA on Windows. Linux also has fallocate(FALLOC_FL_COLLAPSE_SIZE) which lets you actually remove a byte range from the middle of a file, shifting up everything that comes after that byte range. So, in fact, you can "move" bytes in a file, on certain filesystems on Linux (ext4, xfs, among others). Unfortunately, support for sparse files is quite variable across platforms. The macOS equivalent to fallocate(FALLOC_FL_PUNCH_HOLE) if fcntl(F_PUNCHHOLE), but the issue is HFS+ has never supported sparse files. UFS did, but UFS support was dropped from 10.7 onwards. The support is supposed to be back in APFS, but I don't know if it actually works.