3 ms·
I've written a few data processing systems over the years and one of my foundational principles is to eschew complex command line parameters and instead just us
by bane 12y ago
I've written a few data processing systems over the years and one of my foundational principles is to eschew complex command line parameters and instead just use the filesystem to infer things.
For example: building a comic book library? Where do I store the publisher, title and issue metadata? Why not just infer it from the organization of the file system?
D.C.Comics/Batman/The Batman #123.cbz
Marvel/Avengers/Avengers - Ultimate #12.cbz
I once built a text processing and conversion system that used a different scheme.
You mount or copy your text files into a folder "source" and it picks up whatever's in there processes it and dumps the converted result out to "sink".
Later I set it up to have some filters on the data. So "sink" turned into a hierarchy of different pre-filtered results.
sink/Car_Makers
sink/countries/France
sink/countries/England
sink/Truck_Makers
And in the end, only documents that fell into one of those classes appeared in the relevant sink folder. At a glance you'd know if there was something relevant to that class of documents or not just by running "ls" or "dir" on it.
It cut hundreds of hours off of analyzing the results since you'd know a head of time if it was likely you'd find something productive related to one of the output classes.
This work with fuse is really exciting.
Imagine dumping in your text files wherever, fuse mounting it and then just exploring the results in all kinds of useful ways. Maybe you write an "entity extraction" filesystem.
So "Mobydick.txt"
Gets turned into
/mnt/mobydick/characters/ishmael/chapters/1
/mnt/mobydick/characters/ishmael/chapters/4
/mnt/mobydick/characters/ahab/chapters/4
/mnt/mobydick/characters/queequeg/chapters/3
/mnt/mobydick/characters/queequeg/chapters/4
/mnt/mobydick/chapters/4/characters/ishmael
/mnt/mobydick/chapters/4/characters/ahab
/mnt/mobydick/chapters/4/characters/queequeg
/mnt/mobydick/locations/Nantucket/chapters/1
/mnt/mobydick/locations/Nantucket/chapters/4
/mnt/mobydick/locations/Pequod/chapters/4
/mnt/mobydick/chapters/4/locations/Nantucket
/mnt/mobydick/chapters/4/locations/Pequod
/mnt/mobydick/chapters/1/text.txt
/mnt/mobydick/chapters/2/text.txt
/mnt/mobydick/chapters/3/text.txt
/mnt/mobydick/chapters/4/text.txt
And now you can use the fuse filesystem as an analytic tool
- batbomb 12y agothat's called a semantic file system
- bane 12y agoI guess kinda. Except the semantic "files" are just artifacts of some process on the real files that transforms them into the output FS. My understanding of Semantic file systems is that the links between files approximate directory traversal but the files or more or less that normal files you'd have on your file system.
- yoodenvranx 12y agoWhat would be also interesting is to store all book files in a database. Then you add tags with author, year and so on and then you can create several FUSE filesystem like author/year/title or just author/title or author/series/title. So you can browse you collection in different ways. The same would work for images or music. Genre/artist/album/title or year/artist/album
- NoMoreNicksLeft 12y agoI'm using Plex for comics. I don't worry about where the metadata is stored. There's an interface to enter it manually, and an API to write an agent script that grabs the metadata for you. Trying to shoehorn it all into the filesystem as directory names seems... destined for disappointment.
- bane 12y agoOut of curiosity, how are you using plex for comics? The photo gallery feature? I keep asking the developers to add cbz and cbr support but it doesn't seem to be coming.
- NoMoreNicksLeft 12y agoYeh, the photos. The developers are assholes, it's a closed source project... but everything else is so bad it's tough to not go with it. I've also figured out how to do audio streams (radio stations).
- pessimizer 12y agoI've been a little obsessed over these same lines. It seems ultimately to boil down to a tag-based fs (or at least that's been the most productive way to search for people trying to do it.) Here are a couple attempts I know of: http://www.tagsistant.net/ http://www.tagsistant.net/ http://web.mit.edu/6.033/2011/wwwdocs/writing-samples/sbezek_dp1.pdf http://web.mit.edu/6.033/2011/wwwdocs/writing-samples/sbezek...