3 ms·
- Moved files: How to not lose all customized information (rating, playlists, playcount, etc.) when a file gets moved to another place? Many filesystems try to
by minhmeoke 4y ago
- Moved files: How to not lose all customized information (rating, playlists, playcount, etc.) when a file gets moved to another place?
Many filesystems try to solve the same problem (eg: customize the appearance of files in a particular folder [1]). One solution is adding extended file attributes [2], however this might not be supported on all operating systems.
[1] https://en.wikipedia.org/wiki/.DS_Store https://en.wikipedia.org/wiki/.DS_Store
[2] https://en.wikipedia.org/wiki/Extended_file_attributes https://en.wikipedia.org/wiki/Extended_file_attributes
A slower but more portable solution might be content-addressable storage. Basically, create a directory containing just metadata files for each song. Name each file as the SHA256 sum of the associated music file, and put metadata into it in a binary format like flatbuffers [3] or Cap'n Proto [4] or a plaintext format like TOML [5] if you prefer to make the system human-editable at the cost of lower performance. Even after moving a file to another location, the SHA256 sum of the file should not change.
Note that if you have duplicated files, then there might be hash collisions where you'll have to reconcile metadata differences (or you can just merge the metadata together, keeping attributes with the later timestamp). There are various solutions to this as well like building a parallel directory structure which mirrors your music filesystem, but that can get complicated.
[3] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
[4] https://capnproto.org/ https://capnproto.org/
[5] https://toml.io/en/ https://toml.io/en/
- File-Watchers: How to prevent fully indexing the filesystem over and over again and only react to changes?
When first loading a directory of music into the program, build a merkle tree [6] of the files' hashes and save them to the content-addressable storage directory described above if they do not already exist. Once indexing is complete, serialize the merkle trees for each directory as well, this way the next time the program starts, you can just load these up and check for consistency of the files in the background. Then set up FileSystemWatcher [7] to notify you when the contents of a directory changes, and update the metadata files and merkle trees accordingly.
[6] https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree
[7] https://stackoverflow.com/questions/721714/notification-when-a-file-changes https://stackoverflow.com/questions/721714/notification-when...
- sandreas 4y agoWow this is awesome stuff, thank you very much. My current solution / plan is: - Get byte offsets of the audio stream only (e.g. 384, 834882948) - Ignore metadata and build xxhash only over this part - To be faster, build the hash over 5MB in the center - If a new file is indexed, build hash and look it up in the database - If one match is found, assume this is the file - If 0 or more than one, assume new file The edge case of 2 or more with the same hash has never happened. Worked out pretty well so far, although its not perfect. And C# File API provides FileWatcher possibility, so I get notified whenever there is a change. The first run has to be done as full scan, after starting the full scan, a FileWatcher is registered and pushes to the Queue whenever a file has changed. It's far from perfect, but it works.