5 ms·

Hierarchy can't handle most users. There is a fundamental impedance mismatch between what hierarchies facilitate and how users think of their files. Users t

Hierarchy can't handle most users. There is a fundamental impedance mismatch between what hierarchies facilitate and how users think of their files. Users think of their files as having concepts associated with them. e.g. "this file is a photo, about my vacation, and is about scuba" or "this file is a book, about scuba" The user may attempt to express the numerous concepts they associate with their files in terms of a hierarchy, but chances are good they will fail miserably because hierarchies simply aren't up for the task of organizing even moderately diverse data. Is 'vacation' a subset of 'photos', with 'scuba' being a subset of 'vacation'? Where then does their scuba book go? Is the 'scuba' sub-directory something they must create twice, once under 'vacation' and once under 'books'? Inevitably some concepts get duplicated sub-directories across the system, or the user settles on under-organization. This is because hierarchy isn't up to the task of organizing files in the way humans think about files. The answer of course is some form of file tagging, so that 'scuba' can be a single distinct group that overlaps with both 'books' and 'vacation' and 'photos', but isn't a strict subset of any of those.

zrm 8y ago
> The answer of course is some form of file tagging, so that 'scuba' can be a single distinct group that overlaps with both 'books' and 'vacation' and 'photos', but isn't a strict subset of any of those. But that is separate from what file paths are for, which is essentially a semantically meaningful unique identifier. If you want to take a photo with the camera app, crop it with the image editor app, then attach it to an email with the email app, you shouldn't have to search through your photos again the second two times to re-find the one you're already working on. There ought to be something equivalent to copying the file path to the clipboard so that you can reference the file you're already working with in another app.
- darkpuma 8y ago
  > "But that is separate from what file paths are for" This is close to my core point. File paths and/or file hierarchies are not good for file organization. They have been pressed into this role for historic reasons and out of inertia. Papering over the problems of file hierarchies by implementing a good copy-paste system is not a good solution to the fundamental problem. It alleviates some problems, but not all. You can conceivably copy-paste a file you've been working on in one app into another, but what if you want to retrieve a file you last interacted with a year ago? How will you find that file? Under the file hierarchy scheme, in order to retrieve an old file the user must remember the idiosyncrasies of the hierarchy they created ages ago, with many concepts invariably being split into gods know how many subdirectories, OR nearly all of their files sitting in a small handful of directories, unorganized, with little more than the file name to help retrieve the desired file. These are both poor states of affairs. They require the user to store too much extraneous state about the system in their head in order to have any chance of efficiently finding the files they're looking for. Modern systems have begun to improve on this state of affairs by implementing file searching capabilities that function more like internet search engines than traditional hierarchical directory explorers. In theory the user may be able to simply search "scuba" and the black box file search system may use a variety of methods for pulling up a list of files from numerous different subdirectories it believes to be scuba related. Perhaps it pulls in both /vacations/2017/scuba/neat-fish.jpg and /books/scuba/how-to-scuba.epub because both have the string "scuba" in their path. Or perhaps there is some exotic shit going on behind the scenes and an image classifier was previously used to determine the presence of scuba gear in /Pictures/1002389943-984983-198395.jpg This sort of file search engine is fundamentally an imperfect file tagging system. The tagging in this case is being done automatically in a variety of ways the user may or may not be aware of and understand, and may or may not be able to consciously influence and exploit. Let's take a step back for a moment and define the problem: In the mind of the user, there exists a many-to-many mapping between files and concepts. When a user is thinking of a particular file, they have in mind a set of traits about that file. (For example, the user might be thinking of "that picture I took of a strange fish while scuba diving on vacation." The concepts associated with that file might be picture, fish, scuba, vacation.) How do we get that mapping out of the user's head and into the computer, such that it can be used by the computer to retrieve the files a user desires when thinking about a particular set of concepts? That's not yet a question with a definite answer. Automatic tagging of files with concepts the user is anticipated to have should certainly play a role. Image classification is heavy, and therefore has privacy implications since that sort of thing is virtually always offloaded to some corporate server, but when it works well it's great. There are easy sources for automatic tags as well, such as extracting terms out of the file path, or extracting date ranges or even geolocation out of the file metadata. These are great, but typically prove insufficient on their own. Asking the user to manually tag their files with the concepts they associate each file with gets you a very high quality many-to-many mapping between files and concepts, but that is manually intensive. Notoriously so. One solution is to amortize it; during file creation ask the user to tag the file instead of naming the file. That's okay, but still not great. The user might be snapping 10 photos a second and not care to take the time to tag or even name each individual file. How do we meet the needs of that sort of user? A solution I've been exploring is the creation of statistical models that assist the user in manual tagging, to blur the line between automatic and manual tagging. Here is what I mean by that: suppose the user imports 500 new jpg images, with metadata suggesting they were all taken in within a half hour of each other. The user tags ONE of those pictures with "vacation". What is the probability that the remaining pictures in that set are also vacation pictures? Pretty high. (Of course that's a trivial example, the user could just as well import the images all at once in a directory called 'vacation', but this goes back to my earlier point of extracting the many-to-many mapping that exists in the user's head. 'vacation' is merely one tag. By constructing the right statistical models, we can support an arbitrary number of tags.
  - zrm 8y ago
    > You can conceivably copy-paste a file you've been working on in one app into another, but what if you want to retrieve a file you last interacted with a year ago? How will you find that file? With searchable tags, or whatever other thing you can come up with which is better than trying to remember a file path. But that doesn't mean you should get rid of file paths, because they're each good for different things and there is no reason you can't have both. Here's another example. On a desktop you can open a static web page from a local .html file along with all the associated content. You'll have a directory containing an HTML file, maybe some associated .png, .gif, .webm or .js files and so on. A complicated one may have its own hierarchy because it's useful within the context of that document. Then that HTML file is going to have the names of the other files in it, including their relative paths if they're in subdirectories. The same thing can happen with InDesign documents or any other composite document like that. Now how does the user find that HTML file a year later? Using some kind of tags or search function. But the way the HTML file then finds its associated pictures/scripts/videos is using something like filesystem paths. The alternative would be to combine them together into some kind of tarball, which is what some things do (like PDF), but then that's worse -- how do you edit a photo which is embedded in a PDF? Your average image editor will either not support it or will be inclined to do something horrible like rasterize the whole PDF first.
    - darkpuma 8y ago
      As I've detailed here (https://news.ycombinator.com/item?id=19523901 https://news.ycombinator.com/item?id=19523901), the idea is not to remove the traditional hierarchical file system from the low level, but rather to remove trace of it from nearly all human interaction. At the least, interaction with it should be opt-in, not opt-out. You're touching on a broader subject than file organization though, and that is humanist software design. Your static web page example is a good example. That's not humanist software design, it's design that serves the computer. When you save a web page you get an html file and a directory next to it that contains the images, css, etc of that webpage. The user is now burdened with keeping that html file and resource directory next to each other for the rest of the lifespan of that [meta-]file. Consider instead the epub format. It too is one or more html files with images, css, etc. However unlike the traditional local static copy of a web page, all of those files are zipped together and stored as a single file (an epub is a glorified zip file with a special file extension. You can extract them with `unzip`.) This unburdens the user; they no longer have to worry about keeping those files next to each other when copying it, sending it to other people, etc. Epubs are humanist design (and not only for this reason, but I'll save that for another discussion.) Of course inside that epub is a hierarchy of files. And that's fine. A hierarchy of files serves the ebook reader well. While strict hierarchies are poor for user interaction, they're great for software. The details of a file format need not be humanist because the user is not meant to pick it apart. Of course there may be perfectly valid and common use-cases for the user to pick it apart, to pull images out of it for instance. Software that facilitates pulling data out of documents, into a format that is generally useful to the user, is humanist. Consider your common desktop web browser for instance. It allows the user to easily pull an image out of a webpage by right clicking on it and saving it, or perhaps dragging the image out of the browser onto their desktop, or similar modes of interaction. The user needn't save the entire webpage, then poke through a directory filled with numerous files looking for the image they wanted. (At least, not unless the website and browser are conspiring to behave in a manner that is hostile to the user, for example by blocking right clicking....) PDFs and PDF readers are shoddy in numerous ways of course, and strict hierarchical file management has done little to alleviate that.
jpindar 8y ago
True, but hierarchy and tagging can coexist. Sometimes I want to use one, sometimes the other on the same set of things.
- darkpuma 8y ago
  In fact the system I've created as a testbed works by storing a many-to-many tag-filename mapping. However my contention is that hierarchy can be modeled as a subset of tags. Tags are conceptually more powerful and should be the interface exposed to users, not hierarchies. That hierarchies can be expressed by tags is easy to demonstrate. Simply tag each file with all of it's parent directories up to root. Implementing hierarchies in terms of tags is not as computationally efficient, so you wouldn't actually write a filesystem in that way. The point is to create a new interface for humans, not for software. When a human has a dataset that is genuinely best organized as a hierarchy, they can express that hierarchy in terms of tags. Of course traditional file hierarchies implemented in the traditional way, not in terms of tags, must still exist for some purposes. Source code, which requires a deterministic and efficient way to refer to specific other files, is one example. (However the user could still benefit greatly from augmenting the source code tree with tags.) What I'm suggesting is not that hierarchical file systems be removed from operating systems, but rather that they be removed from nearly all user interaction with computers. The "Save as" dialog should not be exposing the hierarchical file system, at least not by default. Ditto the "Open" dialog. Hierarchical file managers like Finder, Explorer, Dolphin, Nautilus, etc should all be revised to be foremost file tagging systems designed to aid the user in managing and querying their tags. The "Desktop" should not be a directory on the filesystem, but rather a tag (or even a complex dynamic query.)
grawprog 8y ago
>Users think of their files as having concepts associated with them. e.g. "this file is a photo, about my vacation, and is about scuba" or "this file is a book, about scuba" Why would someone think of files like that? I don't organize physical objects that way. My books are on my bookshelf and my pictures are in albums or frames. Why would a computer be different. I wouldn't store a book about scuba diving with or relate it to my vacation pictures of me scuba diving in any way in real life, why would I do this on a computer? If I had a collection of books on my computer and pictures from my vacations it would be stored the same way I would do it in real life, keep the books together organized however and keep the pictures together organized however. The topics of my vacation pictures and my books have really nothing to do with eachother, even if they happen to be related. What you describe sounds like the very opposite of how i'd want my files organized.
- darkpuma 8y ago
  > "I wouldn't store a book about scuba diving with or relate it to my vacation pictures of me scuba diving in any way in real life" You don't put every single physical item you have relating to scuba into one big bucket because in the physical world, putting something into one bucket precludes it from being placed into another bucket. When you are organizing things you have to anticipate your most likely queries in advanced. So maybe you have one shelf for science fiction, and another for marine wildlife, and another for all your programming language books. For a while, this works well enough. One book can't go on two shelves, but with few enough books that's not a deal breaker. However, when your personal library grows large enough, this system rapidly becomes impractical. At this point, the librarian resorts to creating a card catalogue, categorizing books by author, subject, etc. Books which have numerous topics have numerous cards in the card catalogue. Since this is the 21st century, card catalogues are now digitized. The digitization of card catalogues saves space, but more importantly it facilitates more powerful queries. You can grab two different stacks of cards, relating to two separate topics, and instantly find the intersection between them. That is, the list of books which appear in both stacks. This greatly reduces the number of cards you must look through to find a book that you're quite certain is in both of the stacks. For instance each stack might have a thousand cards, but the intersection of those two stacks might only be 50 cards. That's 50 cards you have to search through to find your book, instead of a thousand. A file tagging system is a digital card catalogue for your files. And as librarian of your library, you can decide which subjects warrant their own tags and which don't. What you don't have to do is decide ahead of time what sort of queries you want to facilitate. And it's no longer important to keep your collection of books and photographs separated. If you want a book, you intersect whatever you're looking for with the card stack of all books. If you want a photograph, you intersect it with the card stack of all photographs. If you have a book of photographs, it could even be in both, should you want it to be.

zrm 8y ago

> The answer of course is some form of file tagging, so that 'scuba' can be a single distinct group that overlaps with both 'books' and 'vacation' and 'photos', but isn't a strict subset of any of those. But that is separate from what file paths are for, which is essentially a semantically meaningful unique identifier. If you want to take a photo with the camera app, crop it with the image editor app, then attach it to an email with the email app, you shouldn't have to search through your photos again the second two times to re-find the one you're already working on. There ought to be something equivalent to copying the file path to the clipboard so that you can reference the file you're already working with in another app.

darkpuma 8y ago
> "But that is separate from what file paths are for" This is close to my core point. File paths and/or file hierarchies are not good for file organization. They have been pressed into this role for historic reasons and out of inertia. Papering over the problems of file hierarchies by implementing a good copy-paste system is not a good solution to the fundamental problem. It alleviates some problems, but not all. You can conceivably copy-paste a file you've been working on in one app into another, but what if you want to retrieve a file you last interacted with a year ago? How will you find that file? Under the file hierarchy scheme, in order to retrieve an old file the user must remember the idiosyncrasies of the hierarchy they created ages ago, with many concepts invariably being split into gods know how many subdirectories, OR nearly all of their files sitting in a small handful of directories, unorganized, with little more than the file name to help retrieve the desired file. These are both poor states of affairs. They require the user to store too much extraneous state about the system in their head in order to have any chance of efficiently finding the files they're looking for. Modern systems have begun to improve on this state of affairs by implementing file searching capabilities that function more like internet search engines than traditional hierarchical directory explorers. In theory the user may be able to simply search "scuba" and the black box file search system may use a variety of methods for pulling up a list of files from numerous different subdirectories it believes to be scuba related. Perhaps it pulls in both /vacations/2017/scuba/neat-fish.jpg and /books/scuba/how-to-scuba.epub because both have the string "scuba" in their path. Or perhaps there is some exotic shit going on behind the scenes and an image classifier was previously used to determine the presence of scuba gear in /Pictures/1002389943-984983-198395.jpg This sort of file search engine is fundamentally an imperfect file tagging system. The tagging in this case is being done automatically in a variety of ways the user may or may not be aware of and understand, and may or may not be able to consciously influence and exploit. Let's take a step back for a moment and define the problem: In the mind of the user, there exists a many-to-many mapping between files and concepts. When a user is thinking of a particular file, they have in mind a set of traits about that file. (For example, the user might be thinking of "that picture I took of a strange fish while scuba diving on vacation." The concepts associated with that file might be picture, fish, scuba, vacation.) How do we get that mapping out of the user's head and into the computer, such that it can be used by the computer to retrieve the files a user desires when thinking about a particular set of concepts? That's not yet a question with a definite answer. Automatic tagging of files with concepts the user is anticipated to have should certainly play a role. Image classification is heavy, and therefore has privacy implications since that sort of thing is virtually always offloaded to some corporate server, but when it works well it's great. There are easy sources for automatic tags as well, such as extracting terms out of the file path, or extracting date ranges or even geolocation out of the file metadata. These are great, but typically prove insufficient on their own. Asking the user to manually tag their files with the concepts they associate each file with gets you a very high quality many-to-many mapping between files and concepts, but that is manually intensive. Notoriously so. One solution is to amortize it; during file creation ask the user to tag the file instead of naming the file. That's okay, but still not great. The user might be snapping 10 photos a second and not care to take the time to tag or even name each individual file. How do we meet the needs of that sort of user? A solution I've been exploring is the creation of statistical models that assist the user in manual tagging, to blur the line between automatic and manual tagging. Here is what I mean by that: suppose the user imports 500 new jpg images, with metadata suggesting they were all taken in within a half hour of each other. The user tags ONE of those pictures with "vacation". What is the probability that the remaining pictures in that set are also vacation pictures? Pretty high. (Of course that's a trivial example, the user could just as well import the images all at once in a directory called 'vacation', but this goes back to my earlier point of extracting the many-to-many mapping that exists in the user's head. 'vacation' is merely one tag. By constructing the right statistical models, we can support an arbitrary number of tags.
- zrm 8y ago
  > You can conceivably copy-paste a file you've been working on in one app into another, but what if you want to retrieve a file you last interacted with a year ago? How will you find that file? With searchable tags, or whatever other thing you can come up with which is better than trying to remember a file path. But that doesn't mean you should get rid of file paths, because they're each good for different things and there is no reason you can't have both. Here's another example. On a desktop you can open a static web page from a local .html file along with all the associated content. You'll have a directory containing an HTML file, maybe some associated .png, .gif, .webm or .js files and so on. A complicated one may have its own hierarchy because it's useful within the context of that document. Then that HTML file is going to have the names of the other files in it, including their relative paths if they're in subdirectories. The same thing can happen with InDesign documents or any other composite document like that. Now how does the user find that HTML file a year later? Using some kind of tags or search function. But the way the HTML file then finds its associated pictures/scripts/videos is using something like filesystem paths. The alternative would be to combine them together into some kind of tarball, which is what some things do (like PDF), but then that's worse -- how do you edit a photo which is embedded in a PDF? Your average image editor will either not support it or will be inclined to do something horrible like rasterize the whole PDF first.
  - darkpuma 8y ago
    As I've detailed here (https://news.ycombinator.com/item?id=19523901 https://news.ycombinator.com/item?id=19523901), the idea is not to remove the traditional hierarchical file system from the low level, but rather to remove trace of it from nearly all human interaction. At the least, interaction with it should be opt-in, not opt-out. You're touching on a broader subject than file organization though, and that is humanist software design. Your static web page example is a good example. That's not humanist software design, it's design that serves the computer. When you save a web page you get an html file and a directory next to it that contains the images, css, etc of that webpage. The user is now burdened with keeping that html file and resource directory next to each other for the rest of the lifespan of that [meta-]file. Consider instead the epub format. It too is one or more html files with images, css, etc. However unlike the traditional local static copy of a web page, all of those files are zipped together and stored as a single file (an epub is a glorified zip file with a special file extension. You can extract them with `unzip`.) This unburdens the user; they no longer have to worry about keeping those files next to each other when copying it, sending it to other people, etc. Epubs are humanist design (and not only for this reason, but I'll save that for another discussion.) Of course inside that epub is a hierarchy of files. And that's fine. A hierarchy of files serves the ebook reader well. While strict hierarchies are poor for user interaction, they're great for software. The details of a file format need not be humanist because the user is not meant to pick it apart. Of course there may be perfectly valid and common use-cases for the user to pick it apart, to pull images out of it for instance. Software that facilitates pulling data out of documents, into a format that is generally useful to the user, is humanist. Consider your common desktop web browser for instance. It allows the user to easily pull an image out of a webpage by right clicking on it and saving it, or perhaps dragging the image out of the browser onto their desktop, or similar modes of interaction. The user needn't save the entire webpage, then poke through a directory filled with numerous files looking for the image they wanted. (At least, not unless the website and browser are conspiring to behave in a manner that is hostile to the user, for example by blocking right clicking....) PDFs and PDF readers are shoddy in numerous ways of course, and strict hierarchical file management has done little to alleviate that.

jpindar 8y ago

True, but hierarchy and tagging can coexist. Sometimes I want to use one, sometimes the other on the same set of things.

darkpuma 8y ago
In fact the system I've created as a testbed works by storing a many-to-many tag-filename mapping. However my contention is that hierarchy can be modeled as a subset of tags. Tags are conceptually more powerful and should be the interface exposed to users, not hierarchies. That hierarchies can be expressed by tags is easy to demonstrate. Simply tag each file with all of it's parent directories up to root. Implementing hierarchies in terms of tags is not as computationally efficient, so you wouldn't actually write a filesystem in that way. The point is to create a new interface for humans, not for software. When a human has a dataset that is genuinely best organized as a hierarchy, they can express that hierarchy in terms of tags. Of course traditional file hierarchies implemented in the traditional way, not in terms of tags, must still exist for some purposes. Source code, which requires a deterministic and efficient way to refer to specific other files, is one example. (However the user could still benefit greatly from augmenting the source code tree with tags.) What I'm suggesting is not that hierarchical file systems be removed from operating systems, but rather that they be removed from nearly all user interaction with computers. The "Save as" dialog should not be exposing the hierarchical file system, at least not by default. Ditto the "Open" dialog. Hierarchical file managers like Finder, Explorer, Dolphin, Nautilus, etc should all be revised to be foremost file tagging systems designed to aid the user in managing and querying their tags. The "Desktop" should not be a directory on the filesystem, but rather a tag (or even a complex dynamic query.)

grawprog 8y ago

>Users think of their files as having concepts associated with them. e.g. "this file is a photo, about my vacation, and is about scuba" or "this file is a book, about scuba" Why would someone think of files like that? I don't organize physical objects that way. My books are on my bookshelf and my pictures are in albums or frames. Why would a computer be different. I wouldn't store a book about scuba diving with or relate it to my vacation pictures of me scuba diving in any way in real life, why would I do this on a computer? If I had a collection of books on my computer and pictures from my vacations it would be stored the same way I would do it in real life, keep the books together organized however and keep the pictures together organized however. The topics of my vacation pictures and my books have really nothing to do with eachother, even if they happen to be related. What you describe sounds like the very opposite of how i'd want my files organized.

darkpuma 8y ago
> "I wouldn't store a book about scuba diving with or relate it to my vacation pictures of me scuba diving in any way in real life" You don't put every single physical item you have relating to scuba into one big bucket because in the physical world, putting something into one bucket precludes it from being placed into another bucket. When you are organizing things you have to anticipate your most likely queries in advanced. So maybe you have one shelf for science fiction, and another for marine wildlife, and another for all your programming language books. For a while, this works well enough. One book can't go on two shelves, but with few enough books that's not a deal breaker. However, when your personal library grows large enough, this system rapidly becomes impractical. At this point, the librarian resorts to creating a card catalogue, categorizing books by author, subject, etc. Books which have numerous topics have numerous cards in the card catalogue. Since this is the 21st century, card catalogues are now digitized. The digitization of card catalogues saves space, but more importantly it facilitates more powerful queries. You can grab two different stacks of cards, relating to two separate topics, and instantly find the intersection between them. That is, the list of books which appear in both stacks. This greatly reduces the number of cards you must look through to find a book that you're quite certain is in both of the stacks. For instance each stack might have a thousand cards, but the intersection of those two stacks might only be 50 cards. That's 50 cards you have to search through to find your book, instead of a thousand. A file tagging system is a digital card catalogue for your files. And as librarian of your library, you can decide which subjects warrant their own tags and which don't. What you don't have to do is decide ahead of time what sort of queries you want to facilitate. And it's no longer important to keep your collection of books and photographs separated. If you want a book, you intersect whatever you're looking for with the card stack of all books. If you want a photograph, you intersect it with the card stack of all photographs. If you have a book of photographs, it could even be in both, should you want it to be.