50 ms·

I generally agree with the ideas he wrote, such as: > a clean and mutually-beneficial separation between cataloguing and (digital) preservation activities can

by nayuki 6y ago

I generally agree with the ideas he wrote, such as: > a clean and mutually-beneficial separation between cataloguing and (digital) preservation activities can be established > the file's path and/or name are not suitable for use as an identifier; in no small part due to both their transient nature, and inability to be combined with files from other systems which may cause rise to naming conflicts > UUIDs as identifiers; hashes as identifiers In my article https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies https://www.nayuki.io/page/designing-better-file-organizatio... , I start from the same basis and explore deeper structures and issues. I argue that file paths and URLs are indeed bad identifiers, and that the content of a file should determine its name. I propose that a tag consists of some descriptive info like a string, plus a hash reference to the target file being described. The unique feature of my proposition is that the tag is itself a first-class file, which can in turn be tagged or referenced by more metadata.

j-pb 6y ago
That feels like RDF with extra steps ^^' Wouldn't that be the simpler approach? Decomplect the blobs from the metadata? Have a simple triple based graph format, and blobs as simple hash-blob pairs? I mean you even had tuple spaces in there, make them triple spaces and you're halfway there. Bolt on a distributed KV store like BitTorrent and you're finished.
adamretter 6y ago
Thanks for your link to your article, I am a bit short of time this week, but I have had a quick skim and then bookmarked it for reading next week :-)
mceachen 6y ago
In designing PhotoStructure, I've wrestled with many of the issues you outline in your article. * I pretty quickly switched from flat tags to hierarchical tags, though, for expressiveness, presentation, and organization. * I also found it necessary to create a "Asset" that has 1 or more "AssetFile" instances, each with a stable URI and SHA. This scheme supports variant grouping (which is quite common due to backups and jpeg+raw pairs). * The "stable" bit of the URI includes the physical volume UUID, so changing mountpoints don't perturb the underlying AssetFile URI. * I originally used a prefix on the SHA to support multiple algorithms, but found it to be too expensive for subsequent lookups to need to calculate n different SHAs. Instead, there's now a "version" for both Assets and AssetFiles and individual libraries are migrated to new versions automatically (which supports changing SHA algorithms, for example, when that's indicated).
magicalhippo 6y ago
> Example: You have a folder named “School” for all the documents and files related to school activities. You have a folder named “Photos” where you store thousands of pics taken over years. Now you take a couple of photos of activities in volleyball club. Where should you store these photos? Hah this takes me back. My grandpa got his first computer when he was almost 80, running Windows 3.1. He had a non-trivial personal library, and had used index cards[1] to keep track. Fast forward a year, and I'm fixing some issue on his computer. When recreating the issue, he shows me how he saved a copy of each email he sent as a file on disk, so he could easily find it later. He had saved not just one copy, no. He had made a directory tree matching the Dewey decimal system[2], and saved one copy of the email in each relevant subdirectory. You know, like he would write down the details of a book on each relevant index card... So an email with a friend would go into three or four 3-level deep sub directories, depending on the topics discussed. After showing me this, and me standing there in silence, shocked... he turned and asked "is there a better way of doing this?" I had to admit that no, not really. Not in Windows 3.1. [1]: https://en.wikipedia.org/wiki/Index_card https://en.wikipedia.org/wiki/Index_card [2]: https://en.wikipedia.org/wiki/Dewey_Decimal_Classification https://en.wikipedia.org/wiki/Dewey_Decimal_Classification

by nayuki 6y ago

j-pb 6y ago

That feels like RDF with extra steps ^^' Wouldn't that be the simpler approach? Decomplect the blobs from the metadata? Have a simple triple based graph format, and blobs as simple hash-blob pairs? I mean you even had tuple spaces in there, make them triple spaces and you're halfway there. Bolt on a distributed KV store like BitTorrent and you're finished.

adamretter 6y ago

Thanks for your link to your article, I am a bit short of time this week, but I have had a quick skim and then bookmarked it for reading next week :-)

mceachen 6y ago

In designing PhotoStructure, I've wrestled with many of the issues you outline in your article. * I pretty quickly switched from flat tags to hierarchical tags, though, for expressiveness, presentation, and organization. * I also found it necessary to create a "Asset" that has 1 or more "AssetFile" instances, each with a stable URI and SHA. This scheme supports variant grouping (which is quite common due to backups and jpeg+raw pairs). * The "stable" bit of the URI includes the physical volume UUID, so changing mountpoints don't perturb the underlying AssetFile URI. * I originally used a prefix on the SHA to support multiple algorithms, but found it to be too expensive for subsequent lookups to need to calculate n different SHAs. Instead, there's now a "version" for both Assets and AssetFiles and individual libraries are migrated to new versions automatically (which supports changing SHA algorithms, for example, when that's indicated).

magicalhippo 6y ago

> Example: You have a folder named “School” for all the documents and files related to school activities. You have a folder named “Photos” where you store thousands of pics taken over years. Now you take a couple of photos of activities in volleyball club. Where should you store these photos? Hah this takes me back. My grandpa got his first computer when he was almost 80, running Windows 3.1. He had a non-trivial personal library, and had used index cards[1] to keep track. Fast forward a year, and I'm fixing some issue on his computer. When recreating the issue, he shows me how he saved a copy of each email he sent as a file on disk, so he could easily find it later. He had saved not just one copy, no. He had made a directory tree matching the Dewey decimal system[2], and saved one copy of the email in each relevant subdirectory. You know, like he would write down the details of a book on each relevant index card... So an email with a friend would go into three or four 3-level deep sub directories, depending on the topics discussed. After showing me this, and me standing there in silence, shocked... he turned and asked "is there a better way of doing this?" I had to admit that no, not really. Not in Windows 3.1. [1]: https://en.wikipedia.org/wiki/Index_card https://en.wikipedia.org/wiki/Index_card [2]: https://en.wikipedia.org/wiki/Dewey_Decimal_Classification https://en.wikipedia.org/wiki/Dewey_Decimal_Classification