5 ms·

What this boils down to is that he thinks a flat namespace (tags) offers advantages over hierarchical namespaces (tree). They really don't. Once your tag spac

by bborud 5y ago

What this boils down to is that he thinks a flat namespace (tags) offers advantages over hierarchical namespaces (tree). They really don't. Once your tag space grows you will start to struggle with naming, and path-like structures (nested namespaces) start to creep back in. And you are right where you started: paths. The treatment of immutability is too superficial to make any sense of so I don't know what the author is imagining. Ted Nelson has evolved some ideas on this for decades that might be worth knowing about. Some of which have kind of come to pass (if you squint and look at how non-destructive editing tools for video and audio work, for instance). However, very little of Ted's thinking has ever been burdened by usable implementation. The concept of having multiple references to the same file already exists. So what he proposes can be realized with existing file systems just by introducing a different naming scheme and making extensive use of sym-/hard-linking. Yes, a lot of file systems will have terrible lookup and traversal performance, but that problem exists in an orthogonal universe and can be solved. Is, indeed solved, in some fileystems if the marketing blurb doesn't lie. If you think about how you would realize this using existing filesystems, by organizing them differently, the concept isn't as sexy anymore. Because it doesn't really involve a lot of new stuff and you start to see the inconvenience of having to cope with both novelty and problems you didn't have before. The problems someone like me wants solved in filsystems are entirely different, and aren't so much about filesystems as it is about how you make the functionality useful to applications. For instance, there are filesystems that offer snapshot semantics. Including COW-snapshots. This would be useful whenever applications need to do be able to roll back changes, switch between states, do backups while being live etc. Yet I know of no language which has snapshot as part of the standard OS interface. So people generally don't write application that take full advantage of what the underlying system offers.

magicalhippo 5y ago
> And you are right where you started: paths. A big difference is that one can naturally have multiple tags, and an entity could share tags with other entities. Sure you can use hardlinking when it comes to files, but it's tedious and you can't have multiple files hardlinked to the same path.
- bborud 5y ago
  Directories of links. And yes, even with a layer on top of the FS to provide an abstraction (as an API, for instance) so you can build shells and applications, a tag + search based system would quite possibly be tedious to use. I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths.
  - magicalhippo 5y ago
    > Directories of links. Well you'd still have to guarantee uniqueness of the filename within the directory. For example I have several files ala DSC00005.JPG which are not identical, because the camera reset the counter every now and then. > I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths. I agree that 64bit ints is not a stellar solution. If anything it should be an something like UUID, so it can be unique across filesystems, and something the users shouldn't normally have to deal with.
    - bborud 5y ago
      The easiest way to do this is to use the inode for naming the link.
- bin_bash 5y ago
  what's wrong with tags on top of a tree? Do tags even need to be part of the filesystem?
  - oldsecondhand 5y ago
    Maybe not part of the filesystem, but part of the OS API, so that every filebrowser can support it.
    - 5y ago
TheOtherHobbes 5y ago
I'm not sure any of this really addresses the question - which is how do people really use files. IME I have a number of live projects which can contain various numbers of source files, images, web links, PDFs and other documents, text files, and so on. Then there are a number of files I access regularly which may not be associated with a project (like favourite music). Then there's a mountain of data which is just there in case I ever need it. It includes backups of old projects, documents, music and art I keep because I think it's interesting but haven't read yet, web links that are filed and then (sadly...) forgotten, and so on. I don't know how typical this is, and it doesn't matter. Because neither a tag based nor a tree based system address the real issue - which is designing a custom file workflow that collects related references of all kinds, doesn't confuse working data with long-term storage, allows off-site backups, allows collaboration, supports versioning on demand, and also makes it easy to find things. I suppose all of that means some kind of process API which does a lot more than file.open() and file.close(). It could be built on tags, it could be built on trees, it could be built on some combination. Or on something else entirely. The implementation matters a lot less than a set of available features which streamline common tasks in some fairly standardised and effective way.
fluidcruft 5y ago
I think one of the problems is that there are many datasets where objects belong to multiple hierarchies and different hierarchies are more efficient for different tasks. For example I work with medical imaging data. Typically that gets organized around the DICOM object models which even defines multiple structures. Typically that's around the patient/encounter model and slight variations of that and data is stored in a database called a PACS, but working with PACS is extremely difficult because DICOM is optimized for clinical use cases. But there are other ways of organizing the data that are more efficient for other tasks for example for quality improvement or assurance or process monitoring. In fact different users of the data are likely to want views based on different hierarchies. Some software expects certain data layouts etc. There are some efforts to standardize file hierarchies and naming for certain tasks, but perhaps you're not doing that task. You can do things like symbolic links, but trees of symbolic links end up being super fragile in my experience and they're not particularly well supported on some operating systems.
- DerArzt 5y ago
  Couldn't that still be stored in a traditional folder hierarchy, but all in one folder and keep track of the tags related to file names in a sqlite database?
  - fluidcruft 5y ago
    I think the point is that if you're just using the filesystem as blob storage and interacting with a layer database, then you've actually moved beyond the filesystem. Just replace filenames with inodes and that's what you've done.
    - formerly_proven 5y ago
      But that's exactly what many applications have been doing for a long time, they're using S3-like services as CAS or quasi-CAS and generate views for different workflows on the fly from a separate database.
      - 5y ago
horsawlarway 5y ago
Path based file systems take advantage of natural semantics we use for navigation. There is a wonderful overlap between how you navigate the real world, and how you navigate a hierarchical file system. I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags - Tags are manual, often duplicated with slight name changes or variations, hard to discover, and literally worse than a folder hierarchy for discoverability in almost every way. Tags can be nice to have - but only if I also have a path. Otherwise they are utterly inferior.
- formerly_proven 5y ago
  Tags, being one of the most basic implementations of boolean retrieval, tend to suffer from feast-or-famine a lot, at least in my experience. Once you introduce hierarchical tagging, people will just use them like folders with each item having 1.0x tags on average.
- weberer 5y ago
  >I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags If you've ever done online shopping, you probably have. For example, try going on Amazon or Newegg and searching for a GPU. You're shown a sidebar where you can easily filter results be certain tags such as: brand, price range, memory size, core count, in stock, energy star certified, free shipping, etc.
  - horsawlarway 5y ago
    I don't think these systems work nearly as well as you do. Simple example right now: Go on amazon, and search for "intel CPU" - I see the following: 1-16 of 942 results for "intel cpu" Now go back and search for "cpu", then filter by brand "intel" - I see the following: 1-24 of 835 results for "cpu" It turns out the tags are exactly what I said they would be - a hodgepodge of things not correctly applied. For example - searching "intel cpu" actually returns items that include intel CPUs (such as motherboard + cpu bundles) that are missing in just the tagged search. But it's still absolutely a valid result if I was interested in buying a cpu. --- as mostly an aside - I don't really trust Amazon or Newegg to be neutral in their results either, a tagged view is convenient to them as a seller where they can control results.
cptskippy 5y ago
> The concept of having multiple references to the same file already exists. It does and it's really bad IMO. The author's suggestion of unique identifiers though would introduce all sorts of new problems, primarily it would make the transparency problems of existing systems worse. Most applications rely on the location of a file, relative or otherwise to load data (e.g. configuration). That reliance is exploited by software engineers to implement configuration swaps, event processing, and many other features. Referencing files based on UIDs, or a series of tags that aren't guaranteed to be unique or not known to be off limits to regular users, would introduce all manner of complications. I could also see it being terribly easy to introduce bugs loading files using filtered tags. Would applications need to have relative tags to mitigate these problems? Having unique paths works both as a filter for the user and an encapsulation for a system that allows you to localize your concern. Without that encapsulation by default, you will be spending a lot more time and concern dealing with files and tags.
6510 5y ago
There is a lot of stuff that should NEVER appear in a mixed view. (Google is full of examples of that.) Tag coulds and other meta data can still be very useful. The challenge is creating useful tags/meta data automatically. For example a time stamp for every modification or a label for every application that created, modified or loaded the file. Perhaps even the applications you were using when the file was created/modified and the file names of the files loaded into the application. Train some AI to show you files you probably want given your current activity.

by bborud 5y ago

magicalhippo 5y ago

> And you are right where you started: paths. A big difference is that one can naturally have multiple tags, and an entity could share tags with other entities. Sure you can use hardlinking when it comes to files, but it's tedious and you can't have multiple files hardlinked to the same path.

bborud 5y ago
Directories of links. And yes, even with a layer on top of the FS to provide an abstraction (as an API, for instance) so you can build shells and applications, a tag + search based system would quite possibly be tedious to use. I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths.
- magicalhippo 5y ago
  > Directories of links. Well you'd still have to guarantee uniqueness of the filename within the directory. For example I have several files ala DSC00005.JPG which are not identical, because the camera reset the counter every now and then. > I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths. I agree that 64bit ints is not a stellar solution. If anything it should be an something like UUID, so it can be unique across filesystems, and something the users shouldn't normally have to deal with.
  - bborud 5y ago
    The easiest way to do this is to use the inode for naming the link.
bin_bash 5y ago
what's wrong with tags on top of a tree? Do tags even need to be part of the filesystem?
- oldsecondhand 5y ago
  Maybe not part of the filesystem, but part of the OS API, so that every filebrowser can support it.
  - 5y ago

TheOtherHobbes 5y ago

I'm not sure any of this really addresses the question - which is how do people really use files. IME I have a number of live projects which can contain various numbers of source files, images, web links, PDFs and other documents, text files, and so on. Then there are a number of files I access regularly which may not be associated with a project (like favourite music). Then there's a mountain of data which is just there in case I ever need it. It includes backups of old projects, documents, music and art I keep because I think it's interesting but haven't read yet, web links that are filed and then (sadly...) forgotten, and so on. I don't know how typical this is, and it doesn't matter. Because neither a tag based nor a tree based system address the real issue - which is designing a custom file workflow that collects related references of all kinds, doesn't confuse working data with long-term storage, allows off-site backups, allows collaboration, supports versioning on demand, and also makes it easy to find things. I suppose all of that means some kind of process API which does a lot more than file.open() and file.close(). It could be built on tags, it could be built on trees, it could be built on some combination. Or on something else entirely. The implementation matters a lot less than a set of available features which streamline common tasks in some fairly standardised and effective way.

fluidcruft 5y ago

I think one of the problems is that there are many datasets where objects belong to multiple hierarchies and different hierarchies are more efficient for different tasks. For example I work with medical imaging data. Typically that gets organized around the DICOM object models which even defines multiple structures. Typically that's around the patient/encounter model and slight variations of that and data is stored in a database called a PACS, but working with PACS is extremely difficult because DICOM is optimized for clinical use cases. But there are other ways of organizing the data that are more efficient for other tasks for example for quality improvement or assurance or process monitoring. In fact different users of the data are likely to want views based on different hierarchies. Some software expects certain data layouts etc. There are some efforts to standardize file hierarchies and naming for certain tasks, but perhaps you're not doing that task. You can do things like symbolic links, but trees of symbolic links end up being super fragile in my experience and they're not particularly well supported on some operating systems.

DerArzt 5y ago
Couldn't that still be stored in a traditional folder hierarchy, but all in one folder and keep track of the tags related to file names in a sqlite database?
- fluidcruft 5y ago
  I think the point is that if you're just using the filesystem as blob storage and interacting with a layer database, then you've actually moved beyond the filesystem. Just replace filenames with inodes and that's what you've done.
  - formerly_proven 5y ago
    But that's exactly what many applications have been doing for a long time, they're using S3-like services as CAS or quasi-CAS and generate views for different workflows on the fly from a separate database.
    - 5y ago

horsawlarway 5y ago

Path based file systems take advantage of natural semantics we use for navigation. There is a wonderful overlap between how you navigate the real world, and how you navigate a hierarchical file system. I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags - Tags are manual, often duplicated with slight name changes or variations, hard to discover, and literally worse than a folder hierarchy for discoverability in almost every way. Tags can be nice to have - but only if I also have a path. Otherwise they are utterly inferior.

formerly_proven 5y ago
Tags, being one of the most basic implementations of boolean retrieval, tend to suffer from feast-or-famine a lot, at least in my experience. Once you introduce hierarchical tagging, people will just use them like folders with each item having 1.0x tags on average.
weberer 5y ago
>I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags If you've ever done online shopping, you probably have. For example, try going on Amazon or Newegg and searching for a GPU. You're shown a sidebar where you can easily filter results be certain tags such as: brand, price range, memory size, core count, in stock, energy star certified, free shipping, etc.
- horsawlarway 5y ago
  I don't think these systems work nearly as well as you do. Simple example right now: Go on amazon, and search for "intel CPU" - I see the following: 1-16 of 942 results for "intel cpu" Now go back and search for "cpu", then filter by brand "intel" - I see the following: 1-24 of 835 results for "cpu" It turns out the tags are exactly what I said they would be - a hodgepodge of things not correctly applied. For example - searching "intel cpu" actually returns items that include intel CPUs (such as motherboard + cpu bundles) that are missing in just the tagged search. But it's still absolutely a valid result if I was interested in buying a cpu. --- as mostly an aside - I don't really trust Amazon or Newegg to be neutral in their results either, a tagged view is convenient to them as a seller where they can control results.

cptskippy 5y ago

> The concept of having multiple references to the same file already exists. It does and it's really bad IMO. The author's suggestion of unique identifiers though would introduce all sorts of new problems, primarily it would make the transparency problems of existing systems worse. Most applications rely on the location of a file, relative or otherwise to load data (e.g. configuration). That reliance is exploited by software engineers to implement configuration swaps, event processing, and many other features. Referencing files based on UIDs, or a series of tags that aren't guaranteed to be unique or not known to be off limits to regular users, would introduce all manner of complications. I could also see it being terribly easy to introduce bugs loading files using filtered tags. Would applications need to have relative tags to mitigate these problems? Having unique paths works both as a filter for the user and an encapsulation for a system that allows you to localize your concern. Without that encapsulation by default, you will be spending a lot more time and concern dealing with files and tags.

6510 5y ago

There is a lot of stuff that should NEVER appear in a mixed view. (Google is full of examples of that.) Tag coulds and other meta data can still be very useful. The challenge is creating useful tags/meta data automatically. For example a time stamp for every modification or a label for every application that created, modified or loaded the file. Perhaps even the applications you were using when the file was created/modified and the file names of the files loaded into the application. Train some AI to show you files you probably want given your current activity.