6 ms·

Hardly obscure, every modern OS has an equivalent feature, but only OSX and Windows unify it with the regular filesystem API. Streams and resource forks are a

by _wmd 10y ago

Hardly obscure, every modern OS has an equivalent feature, but only OSX and Windows unify it with the regular filesystem API. Streams and resource forks are a play on a now-standard UNIX feature that almost nobody uses because it has a shitty non-file based API that also breaks most tools unless they are specifically aware of them: extended attributes. Resource forks and extended attributes are almost equivalent in every single way, except that extended attributes can only be read/written atomically (limiting their size to strings that will fit in RAM), whereas a fork or stream can be opened like a regular file. Stick that in your pipe and smoke it, UNIX sycophants, another case where Windows is more UNIX than UNIX ;) The file-or-directory vagueness created by the hierarchy of resources buried within a file also more closely maps how the most popular path naming scheme on the planet (URLs) work: an URL can always represent both a file and a collection simultaneously, so I see this as closer to an ideal than the alternative where files can have no children at all. Sadly nobody actually uses these APIs like that, because all our tooling sucks so bad at coping with it. I sometimes wonder what the world would look like if directories on popular operating systems had simply been made 0 byte files

ksherlock 10y ago
Solaris unfies it too. You can even use the runat command to open a shell where extended attributes are exposed as and can be manipulated as regular files. http://docs.oracle.com/cd/E23824_01/html/821-1474/fsattr-5.html http://docs.oracle.com/cd/E23824_01/html/821-1474/fsattr-5.h...
amyjess 10y ago
> feature that almost nobody uses because it has a shitty non-file based API that also breaks most tools unless they are specifically aware of them: extended attributes Mind you, OS X makes extensive use of extended attributes in addition to resource forks (and it's largely deprecated resource forks in favor of app folders). Spend some time poking around Siracusa's reviews (since Tiger); he loves to go into detail about every new way Apple makes use of extended attributes. Also, it's not fair to say that almost nobody uses them. Chrome makes use of extended attributes, as does KDE's metadata system and a few other things. > (limiting their size to strings that will fit in RAM) That's an understatement. The Linux kernel API limits the size of all extended attributes to 64KB, and the most popular filesystems limit them further to 4KB. That's not really comparable to a true fork. ZFS is the exception: its extended attributes are implemented as forks, and the maximum size of an extended attribute is the same as that of a file. Unfortunately, those aren't accessible on ZOL because the kernel won't support it, so you can really only take advantage of it on Solaris/Illumos (and maybe FreeBSD?).
- rayvd 10y ago
  Does OSX use extended attributes to track things like the "color" attribute on a file (that shows up in Finder)? Or is this tracked via the .DS_Store hidden file?
  - 0x0 10y ago
    It uses XAs, check with: mdls some-file-tagged-green (...) kMDItemUserTags = ( Green ) or with xattr: xattr -l some-file-tagged-green (...) com.apple.metadata:_kMDItemUserTags: 00000000 62 70 6C 69 73 74 30 30 A1 01 57 47 72 65 65 6E |bplist00..WGreen|
  - swift 10y ago
    I believe the .DS_Store files are just a fallback so if you are accessing files on a network share or on a file system that doesn't support extended attributes like FAT32, those features can still work. The native implementation on HFS+ uses the extended attributes.
    - 0x0 10y ago
      .DS_Store is where finder saves the folder ui configuation (icon positions, list mode, etc) and these files are present even on native HFS+ (although "hidden" by default like any other unix file name starting with ".") There were some reports online that the future APFS in the 10.12 betas didn't leave .DS_Store files around.
Dylan16807 10y ago
I think if "almost nobody uses it" it's fair to call it obscure. > another case where Windows is more UNIX than UNIX Windows has extended attributes too. Having both features makes it more like a kitchen sink.
- chris_wot 10y ago
  It's used in lots of places. Internet Explorer uses it to save whether a file was downloaded via IE. They are just only useful on NTFS, and often not even then because file hating utilities like Dropbox don't store them. So if you upload a file with an ADS to Dropbox, then copy it back again, you'll have lost that data. One thing I'm not sure about is whether it appears in the file size when using dir. And if you apply a file hashing algorithm to generate a hash and you only use the file attributes, base file name and $DEFAULT data stream then you can append to the file via another data stream. So hash utilities need to be AFS aware to be truly useful in Windows. Unless you are calling data an "attribute" though, it's really a bit of a silly comparison. Literally it's a seperate namespace in which you store data. The standard tools and utilities provided by Windows generally only look at $DEFAULT. The article is correct, git is pretty much doing something very similar, only the data is stored in .git (or specified somewhere else) and you use a tool like git to get access to that data, but you can also dive into the directory directly with any other tool. In Windows you use streams.exe, and it's a. generalised, b. non-portable as it's an intrinsic part of NTFS, and c. denoted as part of the NTFS filename by the delimiter ":", which is a reserved character and documented as such. https://blogs.technet.microsoft.com/askcore/2013/03/24/alternate-data-streams-in-ntfs/ https://blogs.technet.microsoft.com/askcore/2013/03/24/alter...
  - besselheim 10y ago
    Using dir normally does not display anything related to the alternate data streams, but if you use the /r option, their names and sizes will be shown.
  - ymse 10y ago
    > Internet Explorer uses it to save whether a file was downloaded via IE. Wait, but why? Chromium (at least on Linux) uses extended attributes too, but to record the origin and referrer of downloaded files (which can be really useful, once you know about it).
rwmj 10y ago
You missed: - Unix xattrs have a terrible API and awful command line tools: listxattr(2) returning \0-separated character arrays with lists of attributes that are next to impossible to decipher in C? - check! Hiding certain xattrs by default based only on their names? - check! - xattrs have magical qualities based on their names, the kernel version, the kernel configuration, and the filesystem mount options (eg. "security.selinux", "trusted.*") - Some xattrs are \0 terminated (and the APIs set and return the \0 making them very awkward to use from shell scripts), some don't, and some are indeterminate. They can also be binary blobs.
- ivank 10y ago
  Also, add too many xattrs and you can no longer get a list of them: As noted in xattr(7), the VFS imposes a limit of 64 kB on the size of the extended attribute name list returned by listxattr(7). If the total size of attribute names attached to a file exceeds this limit, it is no longer possible to retrieve the list of attribute names. http://man7.org/linux/man-pages/man2/listxattr.2.html http://man7.org/linux/man-pages/man2/listxattr.2.html
- nailer 10y ago
  Is \0 ASCII null?
  - rwmj 10y ago
    Yes. For more details see http://man7.org/linux/man-pages/man2/listxattr.2.html http://man7.org/linux/man-pages/man2/listxattr.2.html
wfunction 10y ago
NTFS has extended attributes.
vocatus_gate 10y ago
How....do you know this kind of stuff? Great read, thanks.

by _wmd 10y ago

ksherlock 10y ago

Solaris unfies it too. You can even use the runat command to open a shell where extended attributes are exposed as and can be manipulated as regular files. http://docs.oracle.com/cd/E23824_01/html/821-1474/fsattr-5.html http://docs.oracle.com/cd/E23824_01/html/821-1474/fsattr-5.h...

amyjess 10y ago

> feature that almost nobody uses because it has a shitty non-file based API that also breaks most tools unless they are specifically aware of them: extended attributes Mind you, OS X makes extensive use of extended attributes in addition to resource forks (and it's largely deprecated resource forks in favor of app folders). Spend some time poking around Siracusa's reviews (since Tiger); he loves to go into detail about every new way Apple makes use of extended attributes. Also, it's not fair to say that almost nobody uses them. Chrome makes use of extended attributes, as does KDE's metadata system and a few other things. > (limiting their size to strings that will fit in RAM) That's an understatement. The Linux kernel API limits the size of all extended attributes to 64KB, and the most popular filesystems limit them further to 4KB. That's not really comparable to a true fork. ZFS is the exception: its extended attributes are implemented as forks, and the maximum size of an extended attribute is the same as that of a file. Unfortunately, those aren't accessible on ZOL because the kernel won't support it, so you can really only take advantage of it on Solaris/Illumos (and maybe FreeBSD?).

rayvd 10y ago
Does OSX use extended attributes to track things like the "color" attribute on a file (that shows up in Finder)? Or is this tracked via the .DS_Store hidden file?
- 0x0 10y ago
  It uses XAs, check with: mdls some-file-tagged-green (...) kMDItemUserTags = ( Green ) or with xattr: xattr -l some-file-tagged-green (...) com.apple.metadata:_kMDItemUserTags: 00000000 62 70 6C 69 73 74 30 30 A1 01 57 47 72 65 65 6E |bplist00..WGreen|
- swift 10y ago
  I believe the .DS_Store files are just a fallback so if you are accessing files on a network share or on a file system that doesn't support extended attributes like FAT32, those features can still work. The native implementation on HFS+ uses the extended attributes.
  - 0x0 10y ago
    .DS_Store is where finder saves the folder ui configuation (icon positions, list mode, etc) and these files are present even on native HFS+ (although "hidden" by default like any other unix file name starting with ".") There were some reports online that the future APFS in the 10.12 betas didn't leave .DS_Store files around.

Dylan16807 10y ago

I think if "almost nobody uses it" it's fair to call it obscure. > another case where Windows is more UNIX than UNIX Windows has extended attributes too. Having both features makes it more like a kitchen sink.

chris_wot 10y ago
It's used in lots of places. Internet Explorer uses it to save whether a file was downloaded via IE. They are just only useful on NTFS, and often not even then because file hating utilities like Dropbox don't store them. So if you upload a file with an ADS to Dropbox, then copy it back again, you'll have lost that data. One thing I'm not sure about is whether it appears in the file size when using dir. And if you apply a file hashing algorithm to generate a hash and you only use the file attributes, base file name and $DEFAULT data stream then you can append to the file via another data stream. So hash utilities need to be AFS aware to be truly useful in Windows. Unless you are calling data an "attribute" though, it's really a bit of a silly comparison. Literally it's a seperate namespace in which you store data. The standard tools and utilities provided by Windows generally only look at $DEFAULT. The article is correct, git is pretty much doing something very similar, only the data is stored in .git (or specified somewhere else) and you use a tool like git to get access to that data, but you can also dive into the directory directly with any other tool. In Windows you use streams.exe, and it's a. generalised, b. non-portable as it's an intrinsic part of NTFS, and c. denoted as part of the NTFS filename by the delimiter ":", which is a reserved character and documented as such. https://blogs.technet.microsoft.com/askcore/2013/03/24/alternate-data-streams-in-ntfs/ https://blogs.technet.microsoft.com/askcore/2013/03/24/alter...
- besselheim 10y ago
  Using dir normally does not display anything related to the alternate data streams, but if you use the /r option, their names and sizes will be shown.
- ymse 10y ago
  > Internet Explorer uses it to save whether a file was downloaded via IE. Wait, but why? Chromium (at least on Linux) uses extended attributes too, but to record the origin and referrer of downloaded files (which can be really useful, once you know about it).

rwmj 10y ago

You missed: - Unix xattrs have a terrible API and awful command line tools: listxattr(2) returning \0-separated character arrays with lists of attributes that are next to impossible to decipher in C? - check! Hiding certain xattrs by default based only on their names? - check! - xattrs have magical qualities based on their names, the kernel version, the kernel configuration, and the filesystem mount options (eg. "security.selinux", "trusted.*") - Some xattrs are \0 terminated (and the APIs set and return the \0 making them very awkward to use from shell scripts), some don't, and some are indeterminate. They can also be binary blobs.

ivank 10y ago
Also, add too many xattrs and you can no longer get a list of them: As noted in xattr(7), the VFS imposes a limit of 64 kB on the size of the extended attribute name list returned by listxattr(7). If the total size of attribute names attached to a file exceeds this limit, it is no longer possible to retrieve the list of attribute names. http://man7.org/linux/man-pages/man2/listxattr.2.html http://man7.org/linux/man-pages/man2/listxattr.2.html
nailer 10y ago
Is \0 ASCII null?
- rwmj 10y ago
  Yes. For more details see http://man7.org/linux/man-pages/man2/listxattr.2.html http://man7.org/linux/man-pages/man2/listxattr.2.html

wfunction 10y ago

NTFS has extended attributes.

vocatus_gate 10y ago

How....do you know this kind of stuff? Great read, thanks.