4 ms·

Where is my file's metadata?

31 points by ingve 3y ago

nullindividual 3y ago
NTFS has Alternate Data Streams[0] as an analog, but rarely leveraged outside of Microsoft’s use…. except for Macs[1] and AFP/SMB on NT, back in the day. Streams[2] from Sysinternals will let you read/write ADS (along with PoSh and other tools). ADS may also be a security risk[3]. [0] https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c54dec26-1551-4d3a-a0ea-4fa40f848eb3 https://learn.microsoft.com/en-us/openspecs/windows_protocol... [1] https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/77a73da6-a70e-45f8-b4bb-51dcd85c8ed1 https://learn.microsoft.com/en-us/openspecs/windows_protocol... [2] https://learn.microsoft.com/en-us/sysinternals/downloads/streams https://learn.microsoft.com/en-us/sysinternals/downloads/str... [3] https://blog.netwrix.com/2022/12/16/alternate_data_stream/ https://blog.netwrix.com/2022/12/16/alternate_data_stream/
- loeg 3y ago
  And of course, Linux has support for extattrs and the common Linux filesystems (ext, xfs, btrfs at least) support them.
  - adrian_b 3y ago
    The first file system with extended file attributes was HPFS, which was introduced in the version 1.2 of the OS/2 operating system, in 1989. HPFS was followed by Silicon Graphics XFS and MS NTFS, both in 1993. After XFS became open source, it was the first Linux file system with support for extattrs, which is why it still has its own commands for handling them, besides those that work with any other Linux file system. Then the rest of the Linux file systems have also adopted this feature, with the main exception of tmpfs, which for many years did not support any extattrs, then it added support only for certain security-related extattrs and only now support for user extattrs has been announced, but with some limitations.
- deleted 3y ago
  [deleted]
js2 3y ago
Discussion seems incomplete without mentioning AppleSingle and AppleDouble which allow the resource fork to be preserved outside of HFS/APFS: https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_fo... http://fileformats.archiveteam.org/wiki/AppleDouble http://fileformats.archiveteam.org/wiki/AppleDouble
didntcheck 3y ago
>When the Mac was being developed in the early 1980s, the majority of file systems were crude and simple I get that this is a Mac-focused site, but isn't starting at the personal computer era missing out a big chunk of history? My understanding was that in the early days of mainframes and then minicomputers, records/files consisting of multiple fields, in a system-standard structure, was the norm, and that Unix was somewhat radical in pushing for the minimalist alternative that "a file is just a bag of bytes [in a hierarchical namespace]". I seem to recall this being discussed in the Unix Haters Handbook, in fact. So while the xattr model may not exactly match the "old way", conceptually it was something of a return to form rather than a wholly novel idea
- somat 3y ago
  A lot of the reason that people liked unix was not that it was "better" than it's minicomputer operating system peers but that it embraced simple and holistic interfaces. it is easy to learn and fun to use. It is a hackers os. But robust... Not exactly. My favorite quote which describes the unix viewpoint perfectly is. "We went to lunch afterward, and I remarked to Dennis that easily half the code I was writing in Multics was error recovery code. He said, "We left all that stuff out. If there's an error, we have this routine called panic(), and when it is called, the machine crashes, and you holler down the hall, 'Hey, reboot it.'" https://multicians.org/unix.html https://multicians.org/unix.html footnote: Really the real reason unix was popular was that it was available. AT&T was banned from the operating system business. so it was very easy to get a copy(with source). Many university computer departments did just this. and all those newfangled CS students started hacking on and graduated familiar with the system and brought it to industry with them.
zeroimpl 3y ago
There's an important difference between metadata owned by a file and metadata that a user/program has attached to a file. Metadata onwed by a file ought to be embedded in the file's data, so that it will be preserved if you copy the file to a different system. This is the case for images, pdf, mp3 etc. Metadata attached to a file is useful to simplify many 3rd-party programs, but should generally not be preserved if you copy the file. Stuff like display attributes for Finder, quarantine attributes for the virus scanner, etc. Each of these requirements deserves a different implementation approach, which is why the current state of affairs makes sense to me.
- adrian_b 3y ago
  I do not agree. Any file copy operation should by default make a perfect copy of the source file, without losing any kind of metadata, because that is the meaning of the word. It is not advertised as a partial copy command. Even when some metadata would be useless of the destination, the purpose of the copy may be to store temporarily the file in another place, with the intention to bring it back later to the original location, when no alteration should have happened to the file. If a complete copy cannot be done, e.g. because the destination file system cannot store some of the metadata, e.g. it has timestamps with a lower precision or it does not store some of the extended attributes, like the Linux tmpfs file system, the copy operation should fail with an explicit error message, like when the destination does not have enough space for all file data. Any file copy command should have options to strip some or all metadata from a file, when that is desired, but that should not be the default option. In the past I had unpleasant surprises with the Linux coreutils or with ssh, which lost silently metadata, even with the appropriate command-line options, which should not have been needed, in the case of coreutils because they had a compilation option to ignore the extended attributes and that option had been used in the binary packages available for certain Linux distributions. When copying over the network, I always use rsync over ssh, because it copies reliably all metadata, even between different operating systems and file systems, e.g. Linux, FreeBSD and Windows.
  - zeroimpl 3y ago
    It's a matter of defaults, and no choice of default will work for everybody. I agree that "rsync --archive" ought to preserve extended attributes, same with creating a tarball. But when copying a file, as opposed to creating a backup/archive, I don't think extended attributes should be included by default. And yes I've also had to compile custom versions of rsync at one point to get a version which understood extended attributes on Mac, but I think that's mostly standardized now. EDIT: Also most programs don't even preserve permissions/ACLs on copy, so not sure why they'd preserve attributes by default.
pilgrim0 3y ago
I think the concept of a file as a bag of bytes is a leaky abstraction. That model is fit within the context of file system implementations, which are concerned with allocation, but unfit for high level file usage, which is concerned with interpretation. Anyone who has written code for handling file uploads knows that identifying a file format happens on a best effort basis, and it’s never 100% guaranteed to work. Even attempting to check if a file is a “plain text” is not guaranteed to work at all times. File extensions cannot be relied upon, and trusting a magic number prefix is also not good, as it can be easily tampered. Hence the bag of bytes abstraction forces the development of context-specific conventions, thus exploding the complexity in the transactional information space. If we account for the Web and APIs, HTML and JSON are probably the most prevalent file formats. Imagine attempting to develop for such ecosystems using files without embedded metadata, e.g a separate payload for <head> and tag attributes — it doesn’t make any sense. My point is: metadata _is_ data, and I disagree completely with the author in regard to side-channels being “where metadata is supposed to be”. Perhaps for _file-system specific_ metadata, sure, but for file-content metadata, hell no. In the end, because of this abstraction, the trust level upon a file’s metadata after the file is transferred is practically zero. I think that consistently structured data _objects_ with embedded and verifiable cryptographic signatures would be a better fit for the interconnected world. The fact we rely so much on webpages and their associated internet domains as a means of structured context and provenance hints at the importance of objects and signatures.

31 points by ingve 3y ago

NTFS has Alternate Data Streams[0] as an analog, but rarely leveraged outside of Microsoft’s use…. except for Macs[1] and AFP/SMB on NT, back in the day. Streams[2] from Sysinternals will let you read/write ADS (along with PoSh and other tools). ADS may also be a security risk[3]. [0] https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c54dec26-1551-4d3a-a0ea-4fa40f848eb3 https://learn.microsoft.com/en-us/openspecs/windows_protocol... [1] https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/77a73da6-a70e-45f8-b4bb-51dcd85c8ed1 https://learn.microsoft.com/en-us/openspecs/windows_protocol... [2] https://learn.microsoft.com/en-us/sysinternals/downloads/streams https://learn.microsoft.com/en-us/sysinternals/downloads/str... [3] https://blog.netwrix.com/2022/12/16/alternate_data_stream/ https://blog.netwrix.com/2022/12/16/alternate_data_stream/

loeg 3y ago
And of course, Linux has support for extattrs and the common Linux filesystems (ext, xfs, btrfs at least) support them.
- adrian_b 3y ago
  The first file system with extended file attributes was HPFS, which was introduced in the version 1.2 of the OS/2 operating system, in 1989. HPFS was followed by Silicon Graphics XFS and MS NTFS, both in 1993. After XFS became open source, it was the first Linux file system with support for extattrs, which is why it still has its own commands for handling them, besides those that work with any other Linux file system. Then the rest of the Linux file systems have also adopted this feature, with the main exception of tmpfs, which for many years did not support any extattrs, then it added support only for certain security-related extattrs and only now support for user extattrs has been announced, but with some limitations.
deleted 3y ago
[deleted]

js2 3y ago

Discussion seems incomplete without mentioning AppleSingle and AppleDouble which allow the resource fork to be preserved outside of HFS/APFS: https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_fo... http://fileformats.archiveteam.org/wiki/AppleDouble http://fileformats.archiveteam.org/wiki/AppleDouble

didntcheck 3y ago

>When the Mac was being developed in the early 1980s, the majority of file systems were crude and simple I get that this is a Mac-focused site, but isn't starting at the personal computer era missing out a big chunk of history? My understanding was that in the early days of mainframes and then minicomputers, records/files consisting of multiple fields, in a system-standard structure, was the norm, and that Unix was somewhat radical in pushing for the minimalist alternative that "a file is just a bag of bytes [in a hierarchical namespace]". I seem to recall this being discussed in the Unix Haters Handbook, in fact. So while the xattr model may not exactly match the "old way", conceptually it was something of a return to form rather than a wholly novel idea

somat 3y ago
A lot of the reason that people liked unix was not that it was "better" than it's minicomputer operating system peers but that it embraced simple and holistic interfaces. it is easy to learn and fun to use. It is a hackers os. But robust... Not exactly. My favorite quote which describes the unix viewpoint perfectly is. "We went to lunch afterward, and I remarked to Dennis that easily half the code I was writing in Multics was error recovery code. He said, "We left all that stuff out. If there's an error, we have this routine called panic(), and when it is called, the machine crashes, and you holler down the hall, 'Hey, reboot it.'" https://multicians.org/unix.html https://multicians.org/unix.html footnote: Really the real reason unix was popular was that it was available. AT&T was banned from the operating system business. so it was very easy to get a copy(with source). Many university computer departments did just this. and all those newfangled CS students started hacking on and graduated familiar with the system and brought it to industry with them.

zeroimpl 3y ago

There's an important difference between metadata owned by a file and metadata that a user/program has attached to a file. Metadata onwed by a file ought to be embedded in the file's data, so that it will be preserved if you copy the file to a different system. This is the case for images, pdf, mp3 etc. Metadata attached to a file is useful to simplify many 3rd-party programs, but should generally not be preserved if you copy the file. Stuff like display attributes for Finder, quarantine attributes for the virus scanner, etc. Each of these requirements deserves a different implementation approach, which is why the current state of affairs makes sense to me.

adrian_b 3y ago
I do not agree. Any file copy operation should by default make a perfect copy of the source file, without losing any kind of metadata, because that is the meaning of the word. It is not advertised as a partial copy command. Even when some metadata would be useless of the destination, the purpose of the copy may be to store temporarily the file in another place, with the intention to bring it back later to the original location, when no alteration should have happened to the file. If a complete copy cannot be done, e.g. because the destination file system cannot store some of the metadata, e.g. it has timestamps with a lower precision or it does not store some of the extended attributes, like the Linux tmpfs file system, the copy operation should fail with an explicit error message, like when the destination does not have enough space for all file data. Any file copy command should have options to strip some or all metadata from a file, when that is desired, but that should not be the default option. In the past I had unpleasant surprises with the Linux coreutils or with ssh, which lost silently metadata, even with the appropriate command-line options, which should not have been needed, in the case of coreutils because they had a compilation option to ignore the extended attributes and that option had been used in the binary packages available for certain Linux distributions. When copying over the network, I always use rsync over ssh, because it copies reliably all metadata, even between different operating systems and file systems, e.g. Linux, FreeBSD and Windows.
- zeroimpl 3y ago
  It's a matter of defaults, and no choice of default will work for everybody. I agree that "rsync --archive" ought to preserve extended attributes, same with creating a tarball. But when copying a file, as opposed to creating a backup/archive, I don't think extended attributes should be included by default. And yes I've also had to compile custom versions of rsync at one point to get a version which understood extended attributes on Mac, but I think that's mostly standardized now. EDIT: Also most programs don't even preserve permissions/ACLs on copy, so not sure why they'd preserve attributes by default.

pilgrim0 3y ago

I think the concept of a file as a bag of bytes is a leaky abstraction. That model is fit within the context of file system implementations, which are concerned with allocation, but unfit for high level file usage, which is concerned with interpretation. Anyone who has written code for handling file uploads knows that identifying a file format happens on a best effort basis, and it’s never 100% guaranteed to work. Even attempting to check if a file is a “plain text” is not guaranteed to work at all times. File extensions cannot be relied upon, and trusting a magic number prefix is also not good, as it can be easily tampered. Hence the bag of bytes abstraction forces the development of context-specific conventions, thus exploding the complexity in the transactional information space. If we account for the Web and APIs, HTML and JSON are probably the most prevalent file formats. Imagine attempting to develop for such ecosystems using files without embedded metadata, e.g a separate payload for <head> and tag attributes — it doesn’t make any sense. My point is: metadata _is_ data, and I disagree completely with the author in regard to side-channels being “where metadata is supposed to be”. Perhaps for _file-system specific_ metadata, sure, but for file-content metadata, hell no. In the end, because of this abstraction, the trust level upon a file’s metadata after the file is transferred is practically zero. I think that consistently structured data _objects_ with embedded and verifiable cryptographic signatures would be a better fit for the interconnected world. The fact we rely so much on webpages and their associated internet domains as a means of structured context and provenance hints at the importance of objects and signatures.