5 ms·

>But fundamentally, there is a mismatch between the narrowness of hierarchies and the rich structure of human knowledge, and the proposed system will not presup

>But fundamentally, there is a mismatch between the narrowness of hierarchies and the rich structure of human knowledge, and the proposed system will not presuppose the features of HFSes. This hits the nail on the head ! All the fileSystems I had to work with are fine as engineering tools. By that I mean using them as an engineer works just fine, their own implementation is off topic. As a user though. What the hell ! I don't want to go to c:/users/me/documents/talks/stockholm2018/draft3 I just want to open my document ! I really hope that someday we expose a document based filesystem to the user. The underlying implementation does not matter, we can always add a layer on top of the hierarchical file system. I just want to be able to display : -all the games installed on my system . -all the pictures -all of my text documents . -all of my pictures of Paris etc

roywiggins 9y ago
I eventually had to use a series of command-line tools to shepherd my badly-organized photo collection into something like a usable state, and it basically involved finding all the JPEGs on my system, de-duplicating them, and dropping them into folders based on their date-taken metadata. It was a huge pain, and entirely an artifact of the tyranny of the folder-based filesystem.
- Symbiote 9y ago
  for i in *.jpg; do date=$(exif $i --machine-readable --tag=0x9003 | cut -d ' ' -f 1 | tr : -) mkdir -p $date mv $i $date done But don't many photo-browsing programs allow browsing by date, or even GPS-tagged location? e.g. on KDE's DigiKam I choose "timeline" view (or map view).
  - tzahola 9y ago
    I like the smiley before mkdir
  - Symbiote 9y ago
    Oh, damn. The formatting was lost, I had line breaks in there!
    - Izkata 9y ago
      Indent with 2 spaces to format code: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
Benjammer 9y ago
How do you make sure things are tagged properly though? A file must reside within a folder, even if it's a default location, which forces a user to think about the folder where the file is stored. With tags, a user could very easily forget one tag on a file, and now any filtering on that tag is never going to be aware of the new file existing. What if you find another picture somewhere from your <Paris> trip, but you forget to add the <Travel> tag? Or you have a tag for "cool architecture pics" or something and you miss tagging one of your Paris pics with this when you upload? There just seems like so much friction in properly keeping tags organized, despite how much extremely better the "read" UI is for someone browsing or searching file collections.
- sli 9y ago
  I would probably use a hybrid of the two. Use the tag system for things I've tagged, and fall back to a regular filesystem for untagged files. That's basically how I have Steam setup. Most of my games are tagged, usually with multiple tags, and all untagged games just go under Games. Once I got everything organized (via automation, not manually -- not necessarily relevant, though), keeping up on tagging new games became easy and quick. I imagine there would be automation tools written for a tagging filesystem, that just "knows" a lot of common software, etc., and can get you started.
- on_and_off 9y ago
  That's a group effort. For picture they could read metadata for something as simple as Paris. The tech is there to search for 'architecture' pics, Google Photos demonstrates it. Of course there is the issue of who can see this data and that's not a small one. The recognition itself could run locally, but training data would still need to be mutualized. And most apps would need to adapt and provide more metadata to your documents
- niftich 9y ago
  Manual tags require a lot of curation and upkeep, but some "tags" are really just restatements of attributes or facts about a file, like search filters, e.g. ("Pictures downloaded from the web on 2018-04-05", "Files created during installation of World of Warcraft", "Files opened in the last two weeks"). In fact, a tag-based document filesystem is largely useless without powerful search, where tag keys and values can be searched at will.
cup-of-tea 9y ago
Didn't Windows try to do this starting with 7? It has "libraries". It was awful and I just wanted the real file system back.
- kijin 9y ago
  Libraries would be a-OK if they were the only abstractions available. You can group your stuff into a number of libraries and call it a day. Unfortunately, Microsoft changed their mind and kinda-deprecated-but-not-completely-eliminated libraries in Windows 10. They also added a whole bunch of similar features that don't fully replace the functionality that libraries used to offer. Now there are at least four different places where the same content might be shown: Quick access, folders under This PC, folders in the actual drives (also under This PC), and Libraries (if you choose to display them). It's an unholy mess that needs registry hacks to clean up.
heavenlyblue 9y ago
Why can't you do the tagging with series of symlinks in one directory instead? It is essentially an index.
- nayuki 9y ago
  Symlinks are brittle when files are renamed and when files move from device to device. Also, symlinks don't solve any of the other problems I'm interested in, such as: Not being forced to make unique file names, being able to find files by hash, being able to find duplicate file content, being able to have tags about tags, and more.
  - deleted 9y ago
    [deleted]
megaman22 9y ago
My problem, at least with Windows, is that it's an unholy mishmash of the two ideas. I really just want something like a Unix filesystem, with a single root. Just with things coming off that root that actually make some goddamn sense and aren't relics from the PDP days. Yet on Windows I have different drives that act as independent file trees, and then I have special folders that are handled somewhat differently, applications that save things in hidden folders that non-power users don't know are there, three or four different cloud storage providers that all need their own special folders, applications that can and cannot be installed in custom locations, weird special-case access control restrictions on some folders, some of the time. And all manner of other abominations. It is far too much work to try to setup document storage in a sane way, and commit to actually doing it, and so my experience is that people increasingly don't even bother to try, and just flood their Downloads and Documents folders.
- mjevans 9y ago
  It doesn't help that Windows /never/ (still hasn't) developed an actual standard library abstraction of what the filesystem is. In the Unix world your standard library either handles all supported filesystems for you, or more often, just provides a working abstraction of that via the kernel's own VFS abstraction of the supported filesystems. Literally, in the UNIX/POSIX world there is exactly one way to open a file for a normal application, irrespective of the storage backing for that file and what filesystem might contain it. (IIRC, it's been a while) On Windows the closest a programmer can get to this is opening up a file via calls specific to NTFS (since it's now the predominant filesystem), FAT (most flash drives), eXFAT (other flash drives), ISO9660(etc)/UDF for optical media, or something else for SMB/CIFS mounts. For many years I've thought MS should just make a VFS that presents everything as (virtually) NTFS, and use that as the standard library abstraction. Which of course has very little relation to actually tagging/organizing/search/recall files in a more human way. The biggest hurdle there is probably getting humans to tag/file things correctly and verbosely enough. MOST humans simply can't do a good job at this. For at least 60%, maybe as high as 80 or 90% of the population, it is /literally/ beyond their technical capability to do a good job (at least without a lot of help).
  - 9y ago
mamcx 9y ago
> The underlying implementation does not matter, we can always add a layer on top of the hierarchical file system. I don't think so. hierarchical (trees) are too limited, and my intuition tell me that this lead to a impedance mismatch. This is the reason the relational model take over, is far more powerfull. To have something comparable, you need full graphs. I think is not coincidence that tag-based filesystem have never take off, the tree-based file system can never be good for it, you need or a graph database or a relational database (or both?). And this probably is only viable on SSD/RAM-alike disks. Finally tags are not enough. You need a way to do groups and alias and groups of groups, plus a way to full-text-search and seriously good metadata alongside the file...
threeseed 9y ago
On OSX these issues have been solved for quite a few years. 1. Press Apple-Space. 2. Type "draft3". 3. Select first item. And for finding "all my pictures" just create a Smart Folder based on a series of rules e.g. "File Type = JPG".
- 3131s 9y ago
  There is TMSU for Linux among others. That's the only semantic file system I've tried, and around 10 years ago, but it was easy to use then.

roywiggins 9y ago

I eventually had to use a series of command-line tools to shepherd my badly-organized photo collection into something like a usable state, and it basically involved finding all the JPEGs on my system, de-duplicating them, and dropping them into folders based on their date-taken metadata. It was a huge pain, and entirely an artifact of the tyranny of the folder-based filesystem.

Symbiote 9y ago
for i in *.jpg; do date=$(exif $i --machine-readable --tag=0x9003 | cut -d ' ' -f 1 | tr : -) mkdir -p $date mv $i $date done But don't many photo-browsing programs allow browsing by date, or even GPS-tagged location? e.g. on KDE's DigiKam I choose "timeline" view (or map view).
- tzahola 9y ago
  I like the smiley before mkdir
- Symbiote 9y ago
  Oh, damn. The formatting was lost, I had line breaks in there!
  - Izkata 9y ago
    Indent with 2 spaces to format code: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc

Benjammer 9y ago

How do you make sure things are tagged properly though? A file must reside within a folder, even if it's a default location, which forces a user to think about the folder where the file is stored. With tags, a user could very easily forget one tag on a file, and now any filtering on that tag is never going to be aware of the new file existing. What if you find another picture somewhere from your <Paris> trip, but you forget to add the <Travel> tag? Or you have a tag for "cool architecture pics" or something and you miss tagging one of your Paris pics with this when you upload? There just seems like so much friction in properly keeping tags organized, despite how much extremely better the "read" UI is for someone browsing or searching file collections.

sli 9y ago
I would probably use a hybrid of the two. Use the tag system for things I've tagged, and fall back to a regular filesystem for untagged files. That's basically how I have Steam setup. Most of my games are tagged, usually with multiple tags, and all untagged games just go under Games. Once I got everything organized (via automation, not manually -- not necessarily relevant, though), keeping up on tagging new games became easy and quick. I imagine there would be automation tools written for a tagging filesystem, that just "knows" a lot of common software, etc., and can get you started.
on_and_off 9y ago
That's a group effort. For picture they could read metadata for something as simple as Paris. The tech is there to search for 'architecture' pics, Google Photos demonstrates it. Of course there is the issue of who can see this data and that's not a small one. The recognition itself could run locally, but training data would still need to be mutualized. And most apps would need to adapt and provide more metadata to your documents
niftich 9y ago
Manual tags require a lot of curation and upkeep, but some "tags" are really just restatements of attributes or facts about a file, like search filters, e.g. ("Pictures downloaded from the web on 2018-04-05", "Files created during installation of World of Warcraft", "Files opened in the last two weeks"). In fact, a tag-based document filesystem is largely useless without powerful search, where tag keys and values can be searched at will.

cup-of-tea 9y ago

Didn't Windows try to do this starting with 7? It has "libraries". It was awful and I just wanted the real file system back.

kijin 9y ago
Libraries would be a-OK if they were the only abstractions available. You can group your stuff into a number of libraries and call it a day. Unfortunately, Microsoft changed their mind and kinda-deprecated-but-not-completely-eliminated libraries in Windows 10. They also added a whole bunch of similar features that don't fully replace the functionality that libraries used to offer. Now there are at least four different places where the same content might be shown: Quick access, folders under This PC, folders in the actual drives (also under This PC), and Libraries (if you choose to display them). It's an unholy mess that needs registry hacks to clean up.

heavenlyblue 9y ago

Why can't you do the tagging with series of symlinks in one directory instead? It is essentially an index.

nayuki 9y ago
Symlinks are brittle when files are renamed and when files move from device to device. Also, symlinks don't solve any of the other problems I'm interested in, such as: Not being forced to make unique file names, being able to find files by hash, being able to find duplicate file content, being able to have tags about tags, and more.
- deleted 9y ago
  [deleted]

megaman22 9y ago

My problem, at least with Windows, is that it's an unholy mishmash of the two ideas. I really just want something like a Unix filesystem, with a single root. Just with things coming off that root that actually make some goddamn sense and aren't relics from the PDP days. Yet on Windows I have different drives that act as independent file trees, and then I have special folders that are handled somewhat differently, applications that save things in hidden folders that non-power users don't know are there, three or four different cloud storage providers that all need their own special folders, applications that can and cannot be installed in custom locations, weird special-case access control restrictions on some folders, some of the time. And all manner of other abominations. It is far too much work to try to setup document storage in a sane way, and commit to actually doing it, and so my experience is that people increasingly don't even bother to try, and just flood their Downloads and Documents folders.

mjevans 9y ago
It doesn't help that Windows /never/ (still hasn't) developed an actual standard library abstraction of what the filesystem is. In the Unix world your standard library either handles all supported filesystems for you, or more often, just provides a working abstraction of that via the kernel's own VFS abstraction of the supported filesystems. Literally, in the UNIX/POSIX world there is exactly one way to open a file for a normal application, irrespective of the storage backing for that file and what filesystem might contain it. (IIRC, it's been a while) On Windows the closest a programmer can get to this is opening up a file via calls specific to NTFS (since it's now the predominant filesystem), FAT (most flash drives), eXFAT (other flash drives), ISO9660(etc)/UDF for optical media, or something else for SMB/CIFS mounts. For many years I've thought MS should just make a VFS that presents everything as (virtually) NTFS, and use that as the standard library abstraction. Which of course has very little relation to actually tagging/organizing/search/recall files in a more human way. The biggest hurdle there is probably getting humans to tag/file things correctly and verbosely enough. MOST humans simply can't do a good job at this. For at least 60%, maybe as high as 80 or 90% of the population, it is /literally/ beyond their technical capability to do a good job (at least without a lot of help).
- 9y ago

mamcx 9y ago

> The underlying implementation does not matter, we can always add a layer on top of the hierarchical file system. I don't think so. hierarchical (trees) are too limited, and my intuition tell me that this lead to a impedance mismatch. This is the reason the relational model take over, is far more powerfull. To have something comparable, you need full graphs. I think is not coincidence that tag-based filesystem have never take off, the tree-based file system can never be good for it, you need or a graph database or a relational database (or both?). And this probably is only viable on SSD/RAM-alike disks. Finally tags are not enough. You need a way to do groups and alias and groups of groups, plus a way to full-text-search and seriously good metadata alongside the file...

threeseed 9y ago

On OSX these issues have been solved for quite a few years. 1. Press Apple-Space. 2. Type "draft3". 3. Select first item. And for finding "all my pictures" just create a Smart Folder based on a series of rules e.g. "File Type = JPG".

3131s 9y ago
There is TMSU for Linux among others. That's the only semantic file system I've tried, and around 10 years ago, but it was easy to use then.