5 ms·

I'm currently working on a file indexer for my pet project tonehub[1] (an audio API like navidrome or audiobookshelf). I thought this is gonna be an easy task,

• samsquire 4y ago
  You might find the idea of "cancellation trees" useful. Create a hierarchy of loop data structure of loop iterating variable and loop limit then cancelling any part of the tree cancels its children. If you have a worker thread that has a hot loop that checks a pthread structure as its limit. You can then send a virtual interrupt to it, or preempt it from running further by setting the loop to its limit. You can see my C, Java and Rust code of this in this repository: https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread It can be used to create extremely responsive software without slowing down the system with cancellation checks. https://github.com/samsquire/ideas4#120-cancellation-trees https://github.com/samsquire/ideas4#120-cancellation-trees https://github.com/samsquire/ideas4#99-register-loop https://github.com/samsquire/ideas4#99-register-loop
  • sandreas 4y ago
    Thank you. This is indeed interesting... Currently, I use the `CancellationTokenSource` / `Task` concept of C# and I'm pretty happy with it, but this is definitely worth reading.
    • samsquire 4y ago
      Would you like to talk more about the things you're working on? I am interested in performance, the architecture of software and solving problems. One problem I have with computer systems is data liveness and synchronization. You want to react to change in many situations but don't want to do things inefficiently such as polling regularly. You kind of want to react to change given an event, when that event happens. So you don't need to poll and compare. You also have the problem of identity, how you map data to other data and keep it in synchronization. If you can capture events at source, then you could do the right behaviour. But it's very hard to capture events at source in modern computing systems as not every API has a callback or event log mechanism.
      • sandreas 4y ago
        Sure, why not... so my pet project is basically for managing my audio files. There already is navidrome[1] and audiobookshelf[2]. They work great so far, but some minor details are kind of annoying... The first milestone will be providing a basic API for my files - the main components of this will be the database (postgres), the API (C# + swashbuckle + JsonApiDotNet + Websockets) and the file indexer (C# HostedService). All parts except the file indexer are pretty much done, but it is a critical component, because it has to be as fast and correct as possible. There are multiple approaches to index files... A best case scenario would be an "import" / "move" of files into a library or repository. That way you would be always up to date and always perfectly sorted. Unfortunately, an import would also be a big amount of work, because analysing the files and getting metadata from online sources is... lets say a huge project. And NOT getting metadata would mean, that I cannot move the files while another app manages the metadata. So I took another path - scanning an existing and well tagged library (that I manage with beets[3] for music and m4b-tool[4] / tone[5] for audio books). My current Idea is to have a file indexer that: - can run on multiple sources - runs one full index scan after starting the app - registers a filesystem watcher for every file source and reacts to events - To ensure, no filesource is blocking others, each source is processed by a fixed batch size and then move on the the next file source - If sources are modified (added, changed, deleted), there is a decision, what to do with already running indexers and registered file watchers (added just go to the queue, changed and deleted cancel already running tasks only for this source) - All files are hashed (content only) to ensure, a change of metadata or tags will not change the hash, and if a file is moved, it will recognize this and update instead of delete and insert The database will contain Tag-Values for every possible value. E.g. File.Location music/album/AC_DC/Back in Black/01 - Hells Bells.mp3 FileTag.Type Artist Tag.Value AC/DC That way I can add a fulltext index on the Tags.Value field containing a searchable value while maintaining the FileTag.Type for recommendations. Let's say I search for `AC/DC`, it will provide an auto-complete for all FileTags.Type values, that show a match + a generic one for searching ALL values: Artist: AC/DC FullText: AC/DC Searching for 2010 will show: Released: 2010 Title: 2010 FullText: 2010 because it contains matches in Releasedate and title. There may be a lot to optimize, but I think my current plan goes pretty well. Let me know what you think about this approach :-) [1]: https://www.navidrome.org/ https://www.navidrome.org/ [2]: https://www.audiobookshelf.org/ https://www.audiobookshelf.org/ [3]: https://beets.io/ https://beets.io/ [4]: https://github.com/sandreas/m4b-tool/ https://github.com/sandreas/m4b-tool/ [5]: https://github.com/sandreas/tone/ https://github.com/sandreas/tone/
• minhmeoke 4y ago
  - Moved files: How to not lose all customized information (rating, playlists, playcount, etc.) when a file gets moved to another place? Many filesystems try to solve the same problem (eg: customize the appearance of files in a particular folder [1]). One solution is adding extended file attributes [2], however this might not be supported on all operating systems. [1] https://en.wikipedia.org/wiki/.DS_Store https://en.wikipedia.org/wiki/.DS_Store [2] https://en.wikipedia.org/wiki/Extended_file_attributes https://en.wikipedia.org/wiki/Extended_file_attributes A slower but more portable solution might be content-addressable storage. Basically, create a directory containing just metadata files for each song. Name each file as the SHA256 sum of the associated music file, and put metadata into it in a binary format like flatbuffers [3] or Cap'n Proto [4] or a plaintext format like TOML [5] if you prefer to make the system human-editable at the cost of lower performance. Even after moving a file to another location, the SHA256 sum of the file should not change. Note that if you have duplicated files, then there might be hash collisions where you'll have to reconcile metadata differences (or you can just merge the metadata together, keeping attributes with the later timestamp). There are various solutions to this as well like building a parallel directory structure which mirrors your music filesystem, but that can get complicated. [3] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ [4] https://capnproto.org/ https://capnproto.org/ [5] https://toml.io/en/ https://toml.io/en/ - File-Watchers: How to prevent fully indexing the filesystem over and over again and only react to changes? When first loading a directory of music into the program, build a merkle tree [6] of the files' hashes and save them to the content-addressable storage directory described above if they do not already exist. Once indexing is complete, serialize the merkle trees for each directory as well, this way the next time the program starts, you can just load these up and check for consistency of the files in the background. Then set up FileSystemWatcher [7] to notify you when the contents of a directory changes, and update the metadata files and merkle trees accordingly. [6] https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree [7] https://stackoverflow.com/questions/721714/notification-when-a-file-changes https://stackoverflow.com/questions/721714/notification-when...
  • sandreas 4y ago
    Wow this is awesome stuff, thank you very much. My current solution / plan is: - Get byte offsets of the audio stream only (e.g. 384, 834882948) - Ignore metadata and build xxhash only over this part - To be faster, build the hash over 5MB in the center - If a new file is indexed, build hash and look it up in the database - If one match is found, assume this is the file - If 0 or more than one, assume new file The edge case of 2 or more with the same hash has never happened. Worked out pretty well so far, although its not perfect. And C# File API provides FileWatcher possibility, so I get notified whenever there is a change. The first run has to be done as full scan, after starting the full scan, a FileWatcher is registered and pushes to the Queue whenever a file has changed. It's far from perfect, but it works.