6 ms·

Enduring file references in macOS

69 points by chmaynard 9y ago

shakna 9y ago
> Bookmarks are fine for app developers, who can make a macOS call which returns the current directory path to any given Bookmark. I'm not fine with it. It's an extra function call, it's more overhead to find and open a file, and it's much slower than placing a pointer to an inode.
- klodolph 9y ago
  Not sure what you're suggesting but the comment seems wrong on multiple levels. First, how do you "place a pointer to an inode"? I'm having trouble dissecting that one. You can store an inode number if you like, but actual inode pointers only work at the filesystem level (are you talking about hard links?) Once you’re storing an inode number on disk, you’re basically replacing the well-tested bookmarks system with your own little hack, and the hack won't even work inside a sandbox (which bookmarks explicitly do). These bookmarked files aren’t typically accessed often, usually only in response to user input, so a small performance difference isn't negligible. So it's not rational to complain about performance here. Either your comment makes no sense at all or I don't understand what you're saying.
klodolph 9y ago
The article hints at this at the end, but note that the Mac has worked this way for a long, long time. You would use an “alias” to store the reference to a file. The alias could be a file itself (via the “make alias” command in the Finder) or you could just store the alias as data somewhere. Mac apps have been working this way since the 1990s no problem.
- kitsunesoba 9y ago
  Yep. I can’t find it now but Apple even had a video that marketed this as an advantage of macOS compared to DOS and early versions of Windows. It handled file moves, renames, etc on open files just fine while DOS/Windows would flip out and sometimes eat your unsaved changes.
  - fiddlerwoaroof 9y ago
    I vaguely recall Windows NT 4 telling me to close the file before deleting or renaming it in Windows Explorer
    - klodolph 9y ago
      This one is due to the major difference between the Unix and Windows approach to files. Windows will enforce access locks (unlike Unix, which only has advisory locks) and this is done by path (unlike Unix, which does it by inode). Windows now allows you to delete an open file, however, the way this works is a bit of a hack. If you try to create a new file with the same name, the operation will fail until the deleted file is closed.
derefr 9y ago
My understanding is that Bookmarks are "capabilities", in the research-OS sense: a Bookmark is a reference to a file, yes, but having said bookmark allows your process to access that file even if your app's sandbox doesn't include it. Notice that as of Sierra, apps can't just let you type in a path to choose a save location, or a library file, or whatever else? Your workflow has to pass through the File Open dialog. That's because the File Open dialog returns a Bookmark—and as long as the app retains the Bookmark, it retains the capability to read from/write to that folder (which it doesn't have by default.)
- Gaelan 9y ago
  This restriction has existed since Lion, and (both then and now) only applies to sandboxed apps.
- jakobegger 9y ago
  The file open dialog does not return a bookmark. It just returns an URL, and your sandbox is extended to get permission to use the file. You can then create a bookmark, if you want to keep using the file across restarts of the app.
g09980 9y ago
So long as we remain wedded to directory paths as a means of referencing files and folders, much of our storage will contain duplicated content. As a counterpoint, there are many situations where duplicating content may be preferred. If I'm pasting something into a document, I want it to remain immutable there without the risk of an external change (by me) altering the document that I assume to remain static.
- sly010 9y ago
  I would add that in general there is no one right abstraction. If you want to be able to change the source file, you are best of using symlinks. If you want to be able to change both files and keep them in sync, you should use a hardlink. If you want to create a true copy and modify them independently, just make a copy. Also worth mentioning that most COW file systems support block level deduplication, so if enabled, two files with the same content will be stored only once.
  - Firadeoclus 9y ago
    Rather than semantically splitting the original content (by either using unidirectional links or getting rid of the connection entirely), it seems to me that keeping the content in a versioned database with bidirectional links is much preferable from the user's point of view. The versioned database takes care of grouping related edits and lets you point at a specific, immutable version. While bidirectional links let you efficiently find all usage sites so that you can make an informed decision as to whether they need updating.
jakobegger 9y ago
Bookmarks are great in theory, but in practice they have a few issues: 1) The API is poorly documented, and behaves differently on different macOS versions 2) Error reporting is really bad. Sometimes “opening” a bookmark fails, and there’s no indication why. The error message suggests that the user should change permissions in Finder, which is incorrect and misleading 3) It’s really easy to forget to “close” bookmarks, leading to leaked kernel resources (which might be one reason for 2) 4) Issues with code signing or the keychain can cause bookmarks to silently fail to open. There are no error messages that suggest this is the case, the only way to fix these problems is to search Google and try random things until it starts working again. Bookmarks, just like the whole sandbox API, are a good idea in theory, but they suffer from poor documentation and extremely poor error reporting, making them unreliable. It’s no wonder developers are reluctant to adopt the sandbox unless they want to distribute their app on the App Store.
- kitsunesoba 9y ago
  I haven’t worked with these APIs for a couple of macOS versions but yeah, this reflects my experience too. It’s enough of a frustration that if your application works with arbitrary files often, it might be necessary to opt out of sandboxing (and the App Store) if you want to ensure a consistent experience for users and don’t have the time or resources to test your way through all possible failures. Maybe this area will get some attention with the rumored upcoming AppKit replacement.
  - jakobegger 9y ago
    I strongly doubt that an AppKit replacement would fix the sandbox issues. Also, to be fair, I think that quality of the sandbox has improved significantly over the years. The remaining issues are in general pretty rare (my apps have thousands of users, and I only get a handful of support requests about these issues per year). The problem is that when these issues pop up, there is nothing I can do to debug them -- aside from asking users "have you tried rebooting your computer?" (which does fix the issues occasionally) Apple really needs to make better error reporting. When opening a security scoped bookmark fails, the system should return a proper error message, or at least log the reason for the failure. And they must absolutely fix their error messages. When the sandbox denies access to a file, AppKit shouldn't suggest changing permissions in the Finder -- that won't help a thing.
PhantomGremlin 9y ago
FYI for those trying to experiment with this ... I tried using the command line to see what's under /.vol and nothing appeared. Which confused me, because I found some old stuff on the web that said I should see "a number" there. So apparently macOS no longer displays a volume number there. StackExchange shows the simple technique: https://superuser.com/questions/163941/what-is-the-vol-directory-for https://superuser.com/questions/163941/what-is-the-vol-direc... You just stat a file to learn both its volume and inode. You can then use that pair to access the file: $ stat /etc/hosts 234881026 23347035 -rw-r--r-- 1 root wheel 0 1035 "Jul 15 10:11:35 2010" "May 30 20:55:10 2010" "May 30 20:55:11 2010" "May 30 20:47:32 2010" 4096 8 0 /etc/hosts $ head -3 /.vol/234881026/23347035 ## # Host Database #
joosters 9y ago
You can’t seriously write an article extolling the virtues of inodes as a way of locating files, without pointing out a very clear weakness: the inode of a file can easily change when an application saves a new version of it! Most apps don’t consider the inode as a user-visible (or user-significant) part of file metadata and so will make no effort to preserve it...
- ridiculous_fish 9y ago
  There's also the reverse issue: a deleted file may have its inode recycled. Linux in particular aggressively recycles inodes.
- klodolph 9y ago
  The article doesn't explain it very well, but a bookmark is not just an inode.
crest 9y ago
I would settle for a decent CLI tool for dealing with macOS alias files.
- deleted 9y ago
  [deleted]
- kotharia 9y ago
  Back when I used OS X, I used this CLI utility for resolving HFS+ aliases to actual paths. https://github.com/lipidity/CLIMac/blob/master/src/readalias.c https://github.com/lipidity/CLIMac/blob/master/src/readalias...
chmaynard 9y ago
See also this follow-up article on the various types of file references available in macOS: https://news.ycombinator.com/item?id=16568103 https://news.ycombinator.com/item?id=16568103
nayuki 9y ago
Quoting the article: > This is reflected in the structure of our Home folders. We tend to put still images in Pictures, video in Movies, mail in Mail, then in Documents we collate content from those folders by duplicating it into finished documents. > So long as we remain wedded to directory paths as a means of referencing files and folders, much of our storage will contain duplicated content. Bookmarks are a central part of a more advanced approach to structuring and organising content which can at last move on from the primitive constraints of MS-DOS. > I think that we will see more, not less, use of Bookmarks as enduring file references in future versions of macOS. They don’t spell the end of inodes, aliases, links, or directory paths, but they need to be made much more accessible to users and sysadmins. On the contrary, I think all of these approaches (duplication, bookmarks, inodes, symlinks) are destined to fail. I believe the problem of file organization cannot be solved adequately in a system of hierarchical mutable folders/files. With regards to duplication, I propose the use of hash references. For example, you can put a picture somewhere on your disk, and let it be retrievable by its SHA-256 hash. Then, when you create a document that includes the picture, simply point to it by hash value - this avoids the need to duplicate the full picture content into the document.
theamk 9y ago
While I think bookmarks have their place, I disagree with the main premise: > This is an extraordinary shortcoming. Imagine losing track of a stack of books when you move them from one room to another! This is is how real life works -- there is no bookmarks in real life! If you had a stack of books, and you move it into the attic, all you your memory ("Bradbury is located near the floor on the left") will not work anymore. If a document was on your table, and you put it in the drawer, and then spilled the ink on the table, then your document will remain unmarked. When applied to files, directories have meaning, and magic tracking links, as described in this article, would be surprising. When you delete old pictures from Photos, do you want some Keynote presentation, which lives in a completely different folder, break? When you move the file from 'current_info/' to 'outdated_info_do_not_use/', do you want old bookmarks to be still valid?
beautifulfreak 9y ago
"Finder aliases have never been fully documented by Apple, but have been found to contain within their opaque data structures inode numbers." I was holding out hope that Apple would publish a tool for working with aliases, or at least a specification of aliases so a 3rd party could design a tool. I have a volume with hundreds of aliases that don't work anymore, because the files they pointed to have moved to a new volume. I wish I could batch edit their paths, using find and replace. Many Apple users have asked for an alias editing tool, as I found when I went searching for one, to no avail - not even a kludgy hack exists. I wonder what risk such a tool would pose. Why doesn't Apple document the structure of aliases? Is it just because they're messy, like photoshop psds, and they don't want anyone to see their inelegance? Or is it because it would open Macs to new kinds of mischief? Aliases are fantastically useful, until they break.

69 points by chmaynard 9y ago

> Bookmarks are fine for app developers, who can make a macOS call which returns the current directory path to any given Bookmark. I'm not fine with it. It's an extra function call, it's more overhead to find and open a file, and it's much slower than placing a pointer to an inode.

klodolph 9y ago
Not sure what you're suggesting but the comment seems wrong on multiple levels. First, how do you "place a pointer to an inode"? I'm having trouble dissecting that one. You can store an inode number if you like, but actual inode pointers only work at the filesystem level (are you talking about hard links?) Once you’re storing an inode number on disk, you’re basically replacing the well-tested bookmarks system with your own little hack, and the hack won't even work inside a sandbox (which bookmarks explicitly do). These bookmarked files aren’t typically accessed often, usually only in response to user input, so a small performance difference isn't negligible. So it's not rational to complain about performance here. Either your comment makes no sense at all or I don't understand what you're saying.

klodolph 9y ago

The article hints at this at the end, but note that the Mac has worked this way for a long, long time. You would use an “alias” to store the reference to a file. The alias could be a file itself (via the “make alias” command in the Finder) or you could just store the alias as data somewhere. Mac apps have been working this way since the 1990s no problem.

kitsunesoba 9y ago
Yep. I can’t find it now but Apple even had a video that marketed this as an advantage of macOS compared to DOS and early versions of Windows. It handled file moves, renames, etc on open files just fine while DOS/Windows would flip out and sometimes eat your unsaved changes.
- fiddlerwoaroof 9y ago
  I vaguely recall Windows NT 4 telling me to close the file before deleting or renaming it in Windows Explorer
  - klodolph 9y ago
    This one is due to the major difference between the Unix and Windows approach to files. Windows will enforce access locks (unlike Unix, which only has advisory locks) and this is done by path (unlike Unix, which does it by inode). Windows now allows you to delete an open file, however, the way this works is a bit of a hack. If you try to create a new file with the same name, the operation will fail until the deleted file is closed.

derefr 9y ago

My understanding is that Bookmarks are "capabilities", in the research-OS sense: a Bookmark is a reference to a file, yes, but having said bookmark allows your process to access that file even if your app's sandbox doesn't include it. Notice that as of Sierra, apps can't just let you type in a path to choose a save location, or a library file, or whatever else? Your workflow has to pass through the File Open dialog. That's because the File Open dialog returns a Bookmark—and as long as the app retains the Bookmark, it retains the capability to read from/write to that folder (which it doesn't have by default.)

Gaelan 9y ago
This restriction has existed since Lion, and (both then and now) only applies to sandboxed apps.
jakobegger 9y ago
The file open dialog does not return a bookmark. It just returns an URL, and your sandbox is extended to get permission to use the file. You can then create a bookmark, if you want to keep using the file across restarts of the app.

g09980 9y ago

So long as we remain wedded to directory paths as a means of referencing files and folders, much of our storage will contain duplicated content. As a counterpoint, there are many situations where duplicating content may be preferred. If I'm pasting something into a document, I want it to remain immutable there without the risk of an external change (by me) altering the document that I assume to remain static.

sly010 9y ago
I would add that in general there is no one right abstraction. If you want to be able to change the source file, you are best of using symlinks. If you want to be able to change both files and keep them in sync, you should use a hardlink. If you want to create a true copy and modify them independently, just make a copy. Also worth mentioning that most COW file systems support block level deduplication, so if enabled, two files with the same content will be stored only once.
- Firadeoclus 9y ago
  Rather than semantically splitting the original content (by either using unidirectional links or getting rid of the connection entirely), it seems to me that keeping the content in a versioned database with bidirectional links is much preferable from the user's point of view. The versioned database takes care of grouping related edits and lets you point at a specific, immutable version. While bidirectional links let you efficiently find all usage sites so that you can make an informed decision as to whether they need updating.

jakobegger 9y ago

Bookmarks are great in theory, but in practice they have a few issues: 1) The API is poorly documented, and behaves differently on different macOS versions 2) Error reporting is really bad. Sometimes “opening” a bookmark fails, and there’s no indication why. The error message suggests that the user should change permissions in Finder, which is incorrect and misleading 3) It’s really easy to forget to “close” bookmarks, leading to leaked kernel resources (which might be one reason for 2) 4) Issues with code signing or the keychain can cause bookmarks to silently fail to open. There are no error messages that suggest this is the case, the only way to fix these problems is to search Google and try random things until it starts working again. Bookmarks, just like the whole sandbox API, are a good idea in theory, but they suffer from poor documentation and extremely poor error reporting, making them unreliable. It’s no wonder developers are reluctant to adopt the sandbox unless they want to distribute their app on the App Store.

kitsunesoba 9y ago
I haven’t worked with these APIs for a couple of macOS versions but yeah, this reflects my experience too. It’s enough of a frustration that if your application works with arbitrary files often, it might be necessary to opt out of sandboxing (and the App Store) if you want to ensure a consistent experience for users and don’t have the time or resources to test your way through all possible failures. Maybe this area will get some attention with the rumored upcoming AppKit replacement.
- jakobegger 9y ago
  I strongly doubt that an AppKit replacement would fix the sandbox issues. Also, to be fair, I think that quality of the sandbox has improved significantly over the years. The remaining issues are in general pretty rare (my apps have thousands of users, and I only get a handful of support requests about these issues per year). The problem is that when these issues pop up, there is nothing I can do to debug them -- aside from asking users "have you tried rebooting your computer?" (which does fix the issues occasionally) Apple really needs to make better error reporting. When opening a security scoped bookmark fails, the system should return a proper error message, or at least log the reason for the failure. And they must absolutely fix their error messages. When the sandbox denies access to a file, AppKit shouldn't suggest changing permissions in the Finder -- that won't help a thing.

PhantomGremlin 9y ago

FYI for those trying to experiment with this ... I tried using the command line to see what's under /.vol and nothing appeared. Which confused me, because I found some old stuff on the web that said I should see "a number" there. So apparently macOS no longer displays a volume number there. StackExchange shows the simple technique: https://superuser.com/questions/163941/what-is-the-vol-directory-for https://superuser.com/questions/163941/what-is-the-vol-direc... You just stat a file to learn both its volume and inode. You can then use that pair to access the file: $ stat /etc/hosts 234881026 23347035 -rw-r--r-- 1 root wheel 0 1035 "Jul 15 10:11:35 2010" "May 30 20:55:10 2010" "May 30 20:55:11 2010" "May 30 20:47:32 2010" 4096 8 0 /etc/hosts $ head -3 /.vol/234881026/23347035 ## # Host Database #

joosters 9y ago

You can’t seriously write an article extolling the virtues of inodes as a way of locating files, without pointing out a very clear weakness: the inode of a file can easily change when an application saves a new version of it! Most apps don’t consider the inode as a user-visible (or user-significant) part of file metadata and so will make no effort to preserve it...

ridiculous_fish 9y ago
There's also the reverse issue: a deleted file may have its inode recycled. Linux in particular aggressively recycles inodes.
klodolph 9y ago
The article doesn't explain it very well, but a bookmark is not just an inode.

crest 9y ago

I would settle for a decent CLI tool for dealing with macOS alias files.

deleted 9y ago
[deleted]
kotharia 9y ago
Back when I used OS X, I used this CLI utility for resolving HFS+ aliases to actual paths. https://github.com/lipidity/CLIMac/blob/master/src/readalias.c https://github.com/lipidity/CLIMac/blob/master/src/readalias...

chmaynard 9y ago

See also this follow-up article on the various types of file references available in macOS: https://news.ycombinator.com/item?id=16568103 https://news.ycombinator.com/item?id=16568103

nayuki 9y ago

Quoting the article: > This is reflected in the structure of our Home folders. We tend to put still images in Pictures, video in Movies, mail in Mail, then in Documents we collate content from those folders by duplicating it into finished documents. > So long as we remain wedded to directory paths as a means of referencing files and folders, much of our storage will contain duplicated content. Bookmarks are a central part of a more advanced approach to structuring and organising content which can at last move on from the primitive constraints of MS-DOS. > I think that we will see more, not less, use of Bookmarks as enduring file references in future versions of macOS. They don’t spell the end of inodes, aliases, links, or directory paths, but they need to be made much more accessible to users and sysadmins. On the contrary, I think all of these approaches (duplication, bookmarks, inodes, symlinks) are destined to fail. I believe the problem of file organization cannot be solved adequately in a system of hierarchical mutable folders/files. With regards to duplication, I propose the use of hash references. For example, you can put a picture somewhere on your disk, and let it be retrievable by its SHA-256 hash. Then, when you create a document that includes the picture, simply point to it by hash value - this avoids the need to duplicate the full picture content into the document.

theamk 9y ago

While I think bookmarks have their place, I disagree with the main premise: > This is an extraordinary shortcoming. Imagine losing track of a stack of books when you move them from one room to another! This is is how real life works -- there is no bookmarks in real life! If you had a stack of books, and you move it into the attic, all you your memory ("Bradbury is located near the floor on the left") will not work anymore. If a document was on your table, and you put it in the drawer, and then spilled the ink on the table, then your document will remain unmarked. When applied to files, directories have meaning, and magic tracking links, as described in this article, would be surprising. When you delete old pictures from Photos, do you want some Keynote presentation, which lives in a completely different folder, break? When you move the file from 'current_info/' to 'outdated_info_do_not_use/', do you want old bookmarks to be still valid?

beautifulfreak 9y ago

"Finder aliases have never been fully documented by Apple, but have been found to contain within their opaque data structures inode numbers." I was holding out hope that Apple would publish a tool for working with aliases, or at least a specification of aliases so a 3rd party could design a tool. I have a volume with hundreds of aliases that don't work anymore, because the files they pointed to have moved to a new volume. I wish I could batch edit their paths, using find and replace. Many Apple users have asked for an alias editing tool, as I found when I went searching for one, to no avail - not even a kludgy hack exists. I wonder what risk such a tool would pose. Why doesn't Apple document the structure of aliases? Is it just because they're messy, like photoshop psds, and they don't want anyone to see their inelegance? Or is it because it would open Macs to new kinds of mischief? Aliases are fantastically useful, until they break.