7 ms·

Alternate parts to the main file are present on several platforms, and have the same problems. They are called resource forks on Mac. On Windows/NTFS they are

Alternate parts to the main file are present on several platforms, and have the same problems. They are called resource forks on Mac. On Windows/NTFS they are called alternate data streams - https://en.wikipedia.org/wiki/Fork_(file_system) https://en.wikipedia.org/wiki/Fork_(file_system) - to use specify a colon and name after the filename (eg example.txt:myads). On Unix, Linux, OS/2 etc you can find extended attributes - https://en.wikipedia.org/wiki/Extended_file_attributes https://en.wikipedia.org/wiki/Extended_file_attributes - which allow storing key value pairs on a file. Restrictions exist and vary. As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that. Trying to execute the file from Explorer then explains that it was downloaded and asks if you really want to proceed. On Linux selinux can store labels in the extended attributes. Older ignorant tools aren't going to know about this, but don't substantially harm anything. Modern tools do know about them and do the right thing. (eg copying a downloaded file elsewhere on Windows will still give the warning). The Linux GNU cp command does require a --preserve xattr flag to copy extended attributes and does not do so by default. Dropbox does support them by default and cross platform.

deleted 11y ago
[deleted]
- cstross 11y ago
  Actually, resource forks go all the way back to the origins of classic MacOS circa 1984-85! And they were a lot more than just extended attributes -- they were a b-tree storage abstraction full of stuff like icons, chunks of 68K machine code, and anything else that simply didn't belong in a linear data file. It's a hold-over from MacOS's very non-UNIX origins, and had the makings of an OO filesystem back in the day -- but then the adoption of NeXTStep -- er, sorry, OSX -- obsoleted it overnight. And now it's basically the veriform appendix of OSX, and leads newbies to jump to this rather odd conclusion (that it's somehow "crazy" to do something in a fundmentally non-UNIXy way).
  - sanderjd 11y ago
    Great comment. Is there anywhere I can read about how it worked and what it was used for before it became a "veriform appendix"? It seems really interesting.
    - kevbin 11y ago
      Inside Macintosh Vol I, II, II: http://www.weihenstephan.org/~michaste/pagetable/mac/Inside_Macintosh.pdf http://www.weihenstephan.org/~michaste/pagetable/mac/Inside_...
      - sanderjd 11y ago
        Awesome, thanks.
    - Lammy 11y ago
      This article by John Siracusa was written right around the time Apple was ditching use of resource forks in OS X and contrasts some of their uses with how it's done today: https://news.ycombinator.com/item?id=10678261 https://news.ycombinator.com/item?id=10678261
  - rogerbinns 11y ago
    MacOS also used to store things like the icon and position on the desktop in the resource forks. As commenters here are making clear, they have been repeatedly implemented across a variety of operating environments because it really is useful being able to store additional information alongside the single stream of bits conventionally thought to make up a file. Even on Unix :-)
  - gherkin0 11y ago
    Yeah. And if I recall correctly, in very old versions of the Mac OS. You actually installed things like fonts into your system by copying the resource fork data from the font file into resource fork of the (singular) "System file" which contained the entire OS (except for the shell).
stefantalpalaru 11y ago
You wouldn't put the entire content of a file in an extended attribute, would you?
- granos 11y ago
  Is that a problem with the fs and its features/implementation or a problem with how some application/developer chose to use those features?
- rbanffy 11y ago
  Fonts, when part of a program, were stored in a "FONT" fork (conversely, icons went into "ICON"). It's natural that font files would have nothing in the data fork and the font definition itself would be under FONT. Full documentation is available at https://developer.apple.com/legacy/library/documentation/mac/pdf/ResEditReference.pdf https://developer.apple.com/legacy/library/documentation/mac...
- rogerbinns 11y ago
  There are various arbitrary restrictions in various filesystems and operating systems. The intention is to store smaller amounts of useful information with the file. Smaller means in the bytes/kilobytes range. For example you wouldn't store multi-gigabyte alternate language versions of a video file in its extended attributes. But you could store URLs of where to get them. Or you could attach authorship, review, dates, keywords and similar metadata which would work for any file type, not just those whose format explicitly has that support. If you want to get a handle on what filesystem design is like then I highly recommend Dominic Giampaolo's book on the design and implementation of the Be filesystem (he wrote BeFS). The book is freely available from his website as http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf and includes information about the design of other filesystems too. It isn't exhaustive, but does give a very good grounding in filesystems and does cover extended attributes.
  - stefantalpalaru 11y ago
    Thanks for the recommendation, that book looks interesting.
CurtHagenlocher 11y ago
Arguably, NTFS implemented alternate data streams primarily to support Mac resource forks.
- rogerbinns 11y ago
  Nope. It was originally done as part of OS/2 (Windows NT was sort of OS/3 - the next version of OS/2) and OS/2 had "extended attributes" - https://en.wikipedia.org/wiki/Extended_file_attributes#OS.2F2 https://en.wikipedia.org/wiki/Extended_file_attributes#OS.2F... - so NT also needed them for compatibility.
  - DHJSH 11y ago
    Thank you! All these Mac people think Apple Invented Everything.
    - michaelhoffman 11y ago
      The resource forks of the Macintosh File System date back to 1984, significantly antedating OS/2. https://en.wikipedia.org/wiki/Macintosh_File_System https://en.wikipedia.org/wiki/Macintosh_File_System
    - duaneb 11y ago
      To be fair, resource forks have been around since HFS first came out. I doubt Apple invented this concept, but they definitely predate MS on this front.
- kevbin 11y ago
  Dave Cutler's team writing NT came from VMS which has multi-version files which are pretty much the same as multi-fork files.
  - 4ad 11y ago
    Not the same thing at all. On VMS, when you opened a file, made changes and saved them, you automatically got a new version. You could open any version using the colon syntax, but it's not the same thing at all as resource forks.
rbanffy 11y ago
BTW, one of the first very popular vulnerabilities of IIS involved making it serve a file's source code rather than executing it (a very convenient way to get database login credentials) through a ::$DATA appended to the URL.
jcsiracusa 11y ago
Apple added support for extended attributes to HFS+ in Mac OS X 10.4 Tiger: http://arstechnica.com/apple/2005/04/macosx-10-4/7/#extended-attributes http://arstechnica.com/apple/2005/04/macosx-10-4/7/#extended... Today, resource forks are actually exposed through the extended attributes interface: http://arstechnica.com/apple/2013/10/os-x-10-9/9/#tags-implementation http://arstechnica.com/apple/2013/10/os-x-10-9/9/#tags-imple... You can see the source code for all this in Apple's Darwin open source repository. Example: http://opensource.apple.com/source/xnu/xnu-2782.40.9/bsd/hfs/hfs_xattr.c http://opensource.apple.com/source/xnu/xnu-2782.40.9/bsd/hfs... Please read this, written by the creator of resource forks, to learn why they exist: http://www.folklore.org/StoryView.py?story=The_Grand_Unified_Model.txt http://www.folklore.org/StoryView.py?story=The_Grand_Unified...
- mapgrep 11y ago
  John, Have you heard any rumblings from within Apple about a new filesystem? Is there any hope?
  - jcsiracusa 11y ago
    There's always hope.
    - spiralganglion 11y ago
      Ding.
chetanahuja 11y ago
Conflating extattrs with HFS style resource forks doesn't really make sense. Extended attributes are just that... attributes. They are not the main "data" of the file. Each and every unix file system has some attributes to begin with (permissions, timestamps etc.) and they vary from filesystem to filesystem (e.g., setuid bit, immutable bit etc.). User supplied attributes just extends this concept in a natural way. All the standard tools expect to be able to call open() on the file and start reading a stream of bytes assuming there's sufficient access. That's what a "file" is. Moreover, it's not just unix as an isolated systems. Those expectations are baked into the structure of the entire Internet. When you receiving a "file" over any medium like email, web etc., you're expecting to receive the aforementioned stream of bytes. The attributes (or extended attributes) are not expected to accompany the file data as a general case. Resource forks on the other hand just completely work against reasonable user expectations. The example given in the OP's post is one such instance. A font file that shows up as having zero bytes to every tool that works with files including tools that expect to transfer "files" over the internet. It's just broken by design.
- mdellabitta 11y ago
  Just a datapoint: Macs have been sending files via the Internet with resource forks for a long time, basically since the invention of the Macintosh: https://en.wikipedia.org/wiki/BinHex https://en.wikipedia.org/wiki/BinHex Resource forks are basically legacy from the original MacOS, and something that's being retained for compatibility, not something that's really a current design. The current replacement of resource forks is bundles, where a directory masquerades as a file in the GUI. There are lots of reasons to hate on HFS+, but I wouldn't consider this the most important one.
- marssaxman 11y ago
  It's not broken, it's just old. It's a legacy technology from the original Macintosh system released in 1984 - before it was even called "Mac OS". Your "reasonable user expectations" are conditioned by three decades of experience with systems that largely didn't exist when the Macintosh resource fork was designed.
  - 11y ago
mappu 11y ago
>As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that. One maybe lesser-known use for ADS: FlylinkDC++ (and derivatives) have an option to store TTH hash data in the file's own ADS, instead of in a central hash store. It means that the hash data could be used by multiple applications, but, it's less I/O efficient to make thousands of these <4KB blocks everywhere.

deleted 11y ago

[deleted]

cstross 11y ago
Actually, resource forks go all the way back to the origins of classic MacOS circa 1984-85! And they were a lot more than just extended attributes -- they were a b-tree storage abstraction full of stuff like icons, chunks of 68K machine code, and anything else that simply didn't belong in a linear data file. It's a hold-over from MacOS's very non-UNIX origins, and had the makings of an OO filesystem back in the day -- but then the adoption of NeXTStep -- er, sorry, OSX -- obsoleted it overnight. And now it's basically the veriform appendix of OSX, and leads newbies to jump to this rather odd conclusion (that it's somehow "crazy" to do something in a fundmentally non-UNIXy way).
- sanderjd 11y ago
  Great comment. Is there anywhere I can read about how it worked and what it was used for before it became a "veriform appendix"? It seems really interesting.
  - kevbin 11y ago
    Inside Macintosh Vol I, II, II: http://www.weihenstephan.org/~michaste/pagetable/mac/Inside_Macintosh.pdf http://www.weihenstephan.org/~michaste/pagetable/mac/Inside_...
    - sanderjd 11y ago
      Awesome, thanks.
  - Lammy 11y ago
    This article by John Siracusa was written right around the time Apple was ditching use of resource forks in OS X and contrasts some of their uses with how it's done today: https://news.ycombinator.com/item?id=10678261 https://news.ycombinator.com/item?id=10678261
- rogerbinns 11y ago
  MacOS also used to store things like the icon and position on the desktop in the resource forks. As commenters here are making clear, they have been repeatedly implemented across a variety of operating environments because it really is useful being able to store additional information alongside the single stream of bits conventionally thought to make up a file. Even on Unix :-)
- gherkin0 11y ago
  Yeah. And if I recall correctly, in very old versions of the Mac OS. You actually installed things like fonts into your system by copying the resource fork data from the font file into resource fork of the (singular) "System file" which contained the entire OS (except for the shell).

stefantalpalaru 11y ago

You wouldn't put the entire content of a file in an extended attribute, would you?

granos 11y ago
Is that a problem with the fs and its features/implementation or a problem with how some application/developer chose to use those features?
rbanffy 11y ago
Fonts, when part of a program, were stored in a "FONT" fork (conversely, icons went into "ICON"). It's natural that font files would have nothing in the data fork and the font definition itself would be under FONT. Full documentation is available at https://developer.apple.com/legacy/library/documentation/mac/pdf/ResEditReference.pdf https://developer.apple.com/legacy/library/documentation/mac...
rogerbinns 11y ago
There are various arbitrary restrictions in various filesystems and operating systems. The intention is to store smaller amounts of useful information with the file. Smaller means in the bytes/kilobytes range. For example you wouldn't store multi-gigabyte alternate language versions of a video file in its extended attributes. But you could store URLs of where to get them. Or you could attach authorship, review, dates, keywords and similar metadata which would work for any file type, not just those whose format explicitly has that support. If you want to get a handle on what filesystem design is like then I highly recommend Dominic Giampaolo's book on the design and implementation of the Be filesystem (he wrote BeFS). The book is freely available from his website as http://www.nobius.org/~dbg/practical-file-system-design.pdf http://www.nobius.org/~dbg/practical-file-system-design.pdf and includes information about the design of other filesystems too. It isn't exhaustive, but does give a very good grounding in filesystems and does cover extended attributes.
- stefantalpalaru 11y ago
  Thanks for the recommendation, that book looks interesting.

CurtHagenlocher 11y ago

Arguably, NTFS implemented alternate data streams primarily to support Mac resource forks.

rogerbinns 11y ago
Nope. It was originally done as part of OS/2 (Windows NT was sort of OS/3 - the next version of OS/2) and OS/2 had "extended attributes" - https://en.wikipedia.org/wiki/Extended_file_attributes#OS.2F2 https://en.wikipedia.org/wiki/Extended_file_attributes#OS.2F... - so NT also needed them for compatibility.
- DHJSH 11y ago
  Thank you! All these Mac people think Apple Invented Everything.
  - michaelhoffman 11y ago
    The resource forks of the Macintosh File System date back to 1984, significantly antedating OS/2. https://en.wikipedia.org/wiki/Macintosh_File_System https://en.wikipedia.org/wiki/Macintosh_File_System
  - duaneb 11y ago
    To be fair, resource forks have been around since HFS first came out. I doubt Apple invented this concept, but they definitely predate MS on this front.
kevbin 11y ago
Dave Cutler's team writing NT came from VMS which has multi-version files which are pretty much the same as multi-fork files.
- 4ad 11y ago
  Not the same thing at all. On VMS, when you opened a file, made changes and saved them, you automatically got a new version. You could open any version using the colon syntax, but it's not the same thing at all as resource forks.

rbanffy 11y ago

BTW, one of the first very popular vulnerabilities of IIS involved making it serve a file's source code rather than executing it (a very convenient way to get database login credentials) through a ::$DATA appended to the URL.

jcsiracusa 11y ago

Apple added support for extended attributes to HFS+ in Mac OS X 10.4 Tiger: http://arstechnica.com/apple/2005/04/macosx-10-4/7/#extended-attributes http://arstechnica.com/apple/2005/04/macosx-10-4/7/#extended... Today, resource forks are actually exposed through the extended attributes interface: http://arstechnica.com/apple/2013/10/os-x-10-9/9/#tags-implementation http://arstechnica.com/apple/2013/10/os-x-10-9/9/#tags-imple... You can see the source code for all this in Apple's Darwin open source repository. Example: http://opensource.apple.com/source/xnu/xnu-2782.40.9/bsd/hfs/hfs_xattr.c http://opensource.apple.com/source/xnu/xnu-2782.40.9/bsd/hfs... Please read this, written by the creator of resource forks, to learn why they exist: http://www.folklore.org/StoryView.py?story=The_Grand_Unified_Model.txt http://www.folklore.org/StoryView.py?story=The_Grand_Unified...

mapgrep 11y ago
John, Have you heard any rumblings from within Apple about a new filesystem? Is there any hope?
- jcsiracusa 11y ago
  There's always hope.
  - spiralganglion 11y ago
    Ding.

chetanahuja 11y ago

Conflating extattrs with HFS style resource forks doesn't really make sense. Extended attributes are just that... attributes. They are not the main "data" of the file. Each and every unix file system has some attributes to begin with (permissions, timestamps etc.) and they vary from filesystem to filesystem (e.g., setuid bit, immutable bit etc.). User supplied attributes just extends this concept in a natural way. All the standard tools expect to be able to call open() on the file and start reading a stream of bytes assuming there's sufficient access. That's what a "file" is. Moreover, it's not just unix as an isolated systems. Those expectations are baked into the structure of the entire Internet. When you receiving a "file" over any medium like email, web etc., you're expecting to receive the aforementioned stream of bytes. The attributes (or extended attributes) are not expected to accompany the file data as a general case. Resource forks on the other hand just completely work against reasonable user expectations. The example given in the OP's post is one such instance. A font file that shows up as having zero bytes to every tool that works with files including tools that expect to transfer "files" over the internet. It's just broken by design.

mdellabitta 11y ago
Just a datapoint: Macs have been sending files via the Internet with resource forks for a long time, basically since the invention of the Macintosh: https://en.wikipedia.org/wiki/BinHex https://en.wikipedia.org/wiki/BinHex Resource forks are basically legacy from the original MacOS, and something that's being retained for compatibility, not something that's really a current design. The current replacement of resource forks is bundles, where a directory masquerades as a file in the GUI. There are lots of reasons to hate on HFS+, but I wouldn't consider this the most important one.
marssaxman 11y ago
It's not broken, it's just old. It's a legacy technology from the original Macintosh system released in 1984 - before it was even called "Mac OS". Your "reasonable user expectations" are conditioned by three decades of experience with systems that largely didn't exist when the Macintosh resource fork was designed.
- 11y ago

mappu 11y ago

>As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that. One maybe lesser-known use for ADS: FlylinkDC++ (and derivatives) have an option to store TTH hash data in the file's own ADS, instead of in a central hash store. It means that the hash data could be used by multiple applications, but, it's less I/O efficient to make thousands of these <4KB blocks everywhere.