4 ms·

I think files are too limited to be the base for a memex. I've been managing a lot of my data recently in my custom built "memex" and have found them a poor abs

I think files are too limited to be the base for a memex. I've been managing a lot of my data recently in my custom built "memex" and have found them a poor abstraction in many cases. An example is my music collection. I have 2000+ songs, of which I have music metadata, metadata for myself (added date, found source), and the music itself. Bundling this into the mp3 is unideal. A json file per song is also not great, doable but overwhelming at 2000+. My ideal is a json file for all of it and then linking to the path of the mp3. The abstraction I've made allows you to pick arbitrary positions in the data "tree" when to break from a file tree abstraction to a intra-file data format (JSON, YAML, etc...) I also think the interoperable argument breaks down when you want to do anything off the beaten path. The author mentions how you might have a folder with a presentation or document. But attaching any kind of nonstandard data for the format is impossible, and requires the format parsing clients to know about it. You're limited by what each format supports and the total number of available, mature formats with mature parsers/editors.

fluidcruft 3y ago
My system is basically to emphasize easy rebuild of a database. So I have directories that contain a text file that can be easily parsed to build databases or process workflows. I have found this works extremely well even in draconian IT environments.
- cobertos 3y ago
  That sounds nice. I've diverged a bit from actual files on disk (even though I wanted my system to stay there for simplicity) into an abstract tree structure with a fs-like API. Though you can get back to plain files by just using a single FSTree (in my system). Would love to hear more about the structure of those plain text files you use for regeneration I think it's very important to have the ability to work directly with files, for draconian environments you describe, but just because it's the simplest, stable, most portable, least-time to _something_ abstraction. It just has to be balanced with more complex functionality files alone can't handle.
donkeybeer 3y ago
Whats overwhelming about the json per file? You need to fill up all that data either way whether you divide it up into a json per mp3 or a single mega json.
- cobertos 3y ago
  Well, with a single json file I can pull it into my text editor (Sublime) and do find and replace (and it's a little nicer than find in all files) and I can do things like code block folding, and probably others Im forgetting. Also, I'm not sure how to `jq` across 2000 files but how to with one json file. I've also heard that the filesystem is not a great place to store larger amounts of files like, say, 500k records? My "memex" tries to use well-known abstractions so that I can do certain tasks in other programs until my "memex" can do it. That and because for quite a few operations, I'd like to simply mount the window of a separate program into my "memex" as a sort of window manager (but deeply integrated with the other stuff the "memex" does, like data and tab management) instead of trying to remake such monolithic, decently-built programs

fluidcruft 3y ago

My system is basically to emphasize easy rebuild of a database. So I have directories that contain a text file that can be easily parsed to build databases or process workflows. I have found this works extremely well even in draconian IT environments.

cobertos 3y ago
That sounds nice. I've diverged a bit from actual files on disk (even though I wanted my system to stay there for simplicity) into an abstract tree structure with a fs-like API. Though you can get back to plain files by just using a single FSTree (in my system). Would love to hear more about the structure of those plain text files you use for regeneration I think it's very important to have the ability to work directly with files, for draconian environments you describe, but just because it's the simplest, stable, most portable, least-time to _something_ abstraction. It just has to be balanced with more complex functionality files alone can't handle.

donkeybeer 3y ago

Whats overwhelming about the json per file? You need to fill up all that data either way whether you divide it up into a json per mp3 or a single mega json.

cobertos 3y ago
Well, with a single json file I can pull it into my text editor (Sublime) and do find and replace (and it's a little nicer than find in all files) and I can do things like code block folding, and probably others Im forgetting. Also, I'm not sure how to `jq` across 2000 files but how to with one json file. I've also heard that the filesystem is not a great place to store larger amounts of files like, say, 500k records? My "memex" tries to use well-known abstractions so that I can do certain tasks in other programs until my "memex" can do it. That and because for quite a few operations, I'd like to simply mount the window of a separate program into my "memex" as a sort of window manager (but deeply integrated with the other stuff the "memex" does, like data and tab management) instead of trying to remake such monolithic, decently-built programs