3 ms·

> why even have a file system? None of this implies that the underlying file systems aren't really useful in this situation. In fact, proper use of a file syst

> why even have a file system? None of this implies that the underlying file systems aren't really useful in this situation. In fact, proper use of a file system would include separation of the media indexes from the media themselves, so that if you want to migrate to a better index, you can do so easily.

sparkie 12y ago
It's worth questioning why "proper use" of the file system is never done in practice. The filesystem is the wrong abstraction for this kind of problem, purely because it only offers one-to-many relationships, bar hacks like symbolic links which are rarely used where they'd be suitable, and because it's just simpler to dump metadata into a "container" with the media, and call it a file. Nearly all real-world data has many-to-many relationships - for example, in music, we have several artists for the same song, and of course, artists perform multiple songs. The container formats we use for music are really shoddy and try to force these organic relationships into a limited set of tags, and then our filesystem doesn't even have knowledge of this information unless we add plugins for specific file formats - another messy area. MusicBrainz and similar services organize music "the right way" in a RDBMS, and also includes support for things like multi-language tagging, which is sorely lacking in our filesystems. Instead of trying to come up with "one name to rule them all" for a piece of media, each object is just given an identity, a uuid, then all the metadata is related to the uuid. If we wanted to keep metadata and media separate, the obvious solution here is to dump the media in blobs in the same database - give each piece of media a uuid, and create a new relationship table to link the uuid to its metadata. I'd personally like this solution for my music. To do the same for files, we'd have an explosion of symbolic links to map the relationships, and no tool to really navigate through them effectively because we're missing a query language. If we sat these indexes on the filesystem, we'd have a big loss of performance because of the extra layers of indirection and searching, for which no optimization is done. A filesystem is really just a limited kind of database, but where we have a more powerful tool available, why not use it? Well, one reason is backward compatibility - our programs are written to look for files on filesystems, rather than streams from abstract sources. Perhaps what would be ideal in this kind of situation would be a FUSE layer which can expose the MusicBrainz database as a filesystem, but the underlying storage be the postgres db.
- tim333 12y ago
  >The filesystem is the wrong abstraction for this kind of problem From my experience having moved from PC to Mac, I find iPhoto trying to hide my files and abstract things away a pain in the arse. On the old computer I had them in folders labelled 2012, 2013 etc. I now want to move some previous years to another disk to save memory and it's hard to figure that now. Previously I would just have dragged a folder. I guess my point is that fancy database structures may be hard for the human brain to conceptualise.
- vezzy-fnord 12y ago
  Well, it's true that symlinks are a hack, but there is a proper solution to this problem that fits very neatly within the file system model: namespaces (http://www.cs.bell-labs.com/sys/doc/names.html http://www.cs.bell-labs.com/sys/doc/names.html).

sparkie 12y ago

It's worth questioning why "proper use" of the file system is never done in practice. The filesystem is the wrong abstraction for this kind of problem, purely because it only offers one-to-many relationships, bar hacks like symbolic links which are rarely used where they'd be suitable, and because it's just simpler to dump metadata into a "container" with the media, and call it a file. Nearly all real-world data has many-to-many relationships - for example, in music, we have several artists for the same song, and of course, artists perform multiple songs. The container formats we use for music are really shoddy and try to force these organic relationships into a limited set of tags, and then our filesystem doesn't even have knowledge of this information unless we add plugins for specific file formats - another messy area. MusicBrainz and similar services organize music "the right way" in a RDBMS, and also includes support for things like multi-language tagging, which is sorely lacking in our filesystems. Instead of trying to come up with "one name to rule them all" for a piece of media, each object is just given an identity, a uuid, then all the metadata is related to the uuid. If we wanted to keep metadata and media separate, the obvious solution here is to dump the media in blobs in the same database - give each piece of media a uuid, and create a new relationship table to link the uuid to its metadata. I'd personally like this solution for my music. To do the same for files, we'd have an explosion of symbolic links to map the relationships, and no tool to really navigate through them effectively because we're missing a query language. If we sat these indexes on the filesystem, we'd have a big loss of performance because of the extra layers of indirection and searching, for which no optimization is done. A filesystem is really just a limited kind of database, but where we have a more powerful tool available, why not use it? Well, one reason is backward compatibility - our programs are written to look for files on filesystems, rather than streams from abstract sources. Perhaps what would be ideal in this kind of situation would be a FUSE layer which can expose the MusicBrainz database as a filesystem, but the underlying storage be the postgres db.

tim333 12y ago
>The filesystem is the wrong abstraction for this kind of problem From my experience having moved from PC to Mac, I find iPhoto trying to hide my files and abstract things away a pain in the arse. On the old computer I had them in folders labelled 2012, 2013 etc. I now want to move some previous years to another disk to save memory and it's hard to figure that now. Previously I would just have dragged a folder. I guess my point is that fancy database structures may be hard for the human brain to conceptualise.
vezzy-fnord 12y ago
Well, it's true that symlinks are a hack, but there is a proper solution to this problem that fits very neatly within the file system model: namespaces (http://www.cs.bell-labs.com/sys/doc/names.html http://www.cs.bell-labs.com/sys/doc/names.html).