5 ms·

You might find the idea of "cancellation trees" useful. Create a hierarchy of loop data structure of loop iterating variable and loop limit then cancelling any

You might find the idea of "cancellation trees" useful. Create a hierarchy of loop data structure of loop iterating variable and loop limit then cancelling any part of the tree cancels its children. If you have a worker thread that has a hot loop that checks a pthread structure as its limit. You can then send a virtual interrupt to it, or preempt it from running further by setting the loop to its limit. You can see my C, Java and Rust code of this in this repository: https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread It can be used to create extremely responsive software without slowing down the system with cancellation checks. https://github.com/samsquire/ideas4#120-cancellation-trees https://github.com/samsquire/ideas4#120-cancellation-trees https://github.com/samsquire/ideas4#99-register-loop https://github.com/samsquire/ideas4#99-register-loop

sandreas 4y ago
Thank you. This is indeed interesting... Currently, I use the `CancellationTokenSource` / `Task` concept of C# and I'm pretty happy with it, but this is definitely worth reading.
- samsquire 4y ago
  Would you like to talk more about the things you're working on? I am interested in performance, the architecture of software and solving problems. One problem I have with computer systems is data liveness and synchronization. You want to react to change in many situations but don't want to do things inefficiently such as polling regularly. You kind of want to react to change given an event, when that event happens. So you don't need to poll and compare. You also have the problem of identity, how you map data to other data and keep it in synchronization. If you can capture events at source, then you could do the right behaviour. But it's very hard to capture events at source in modern computing systems as not every API has a callback or event log mechanism.
  - sandreas 4y ago
    Sure, why not... so my pet project is basically for managing my audio files. There already is navidrome[1] and audiobookshelf[2]. They work great so far, but some minor details are kind of annoying... The first milestone will be providing a basic API for my files - the main components of this will be the database (postgres), the API (C# + swashbuckle + JsonApiDotNet + Websockets) and the file indexer (C# HostedService). All parts except the file indexer are pretty much done, but it is a critical component, because it has to be as fast and correct as possible. There are multiple approaches to index files... A best case scenario would be an "import" / "move" of files into a library or repository. That way you would be always up to date and always perfectly sorted. Unfortunately, an import would also be a big amount of work, because analysing the files and getting metadata from online sources is... lets say a huge project. And NOT getting metadata would mean, that I cannot move the files while another app manages the metadata. So I took another path - scanning an existing and well tagged library (that I manage with beets[3] for music and m4b-tool[4] / tone[5] for audio books). My current Idea is to have a file indexer that: - can run on multiple sources - runs one full index scan after starting the app - registers a filesystem watcher for every file source and reacts to events - To ensure, no filesource is blocking others, each source is processed by a fixed batch size and then move on the the next file source - If sources are modified (added, changed, deleted), there is a decision, what to do with already running indexers and registered file watchers (added just go to the queue, changed and deleted cancel already running tasks only for this source) - All files are hashed (content only) to ensure, a change of metadata or tags will not change the hash, and if a file is moved, it will recognize this and update instead of delete and insert The database will contain Tag-Values for every possible value. E.g. File.Location music/album/AC_DC/Back in Black/01 - Hells Bells.mp3 FileTag.Type Artist Tag.Value AC/DC That way I can add a fulltext index on the Tags.Value field containing a searchable value while maintaining the FileTag.Type for recommendations. Let's say I search for `AC/DC`, it will provide an auto-complete for all FileTags.Type values, that show a match + a generic one for searching ALL values: Artist: AC/DC FullText: AC/DC Searching for 2010 will show: Released: 2010 Title: 2010 FullText: 2010 because it contains matches in Releasedate and title. There may be a lot to optimize, but I think my current plan goes pretty well. Let me know what you think about this approach :-) [1]: https://www.navidrome.org/ https://www.navidrome.org/ [2]: https://www.audiobookshelf.org/ https://www.audiobookshelf.org/ [3]: https://beets.io/ https://beets.io/ [4]: https://github.com/sandreas/m4b-tool/ https://github.com/sandreas/m4b-tool/ [5]: https://github.com/sandreas/tone/ https://github.com/sandreas/tone/

sandreas 4y ago

Thank you. This is indeed interesting... Currently, I use the `CancellationTokenSource` / `Task` concept of C# and I'm pretty happy with it, but this is definitely worth reading.

samsquire 4y ago
Would you like to talk more about the things you're working on? I am interested in performance, the architecture of software and solving problems. One problem I have with computer systems is data liveness and synchronization. You want to react to change in many situations but don't want to do things inefficiently such as polling regularly. You kind of want to react to change given an event, when that event happens. So you don't need to poll and compare. You also have the problem of identity, how you map data to other data and keep it in synchronization. If you can capture events at source, then you could do the right behaviour. But it's very hard to capture events at source in modern computing systems as not every API has a callback or event log mechanism.
- sandreas 4y ago
  Sure, why not... so my pet project is basically for managing my audio files. There already is navidrome[1] and audiobookshelf[2]. They work great so far, but some minor details are kind of annoying... The first milestone will be providing a basic API for my files - the main components of this will be the database (postgres), the API (C# + swashbuckle + JsonApiDotNet + Websockets) and the file indexer (C# HostedService). All parts except the file indexer are pretty much done, but it is a critical component, because it has to be as fast and correct as possible. There are multiple approaches to index files... A best case scenario would be an "import" / "move" of files into a library or repository. That way you would be always up to date and always perfectly sorted. Unfortunately, an import would also be a big amount of work, because analysing the files and getting metadata from online sources is... lets say a huge project. And NOT getting metadata would mean, that I cannot move the files while another app manages the metadata. So I took another path - scanning an existing and well tagged library (that I manage with beets[3] for music and m4b-tool[4] / tone[5] for audio books). My current Idea is to have a file indexer that: - can run on multiple sources - runs one full index scan after starting the app - registers a filesystem watcher for every file source and reacts to events - To ensure, no filesource is blocking others, each source is processed by a fixed batch size and then move on the the next file source - If sources are modified (added, changed, deleted), there is a decision, what to do with already running indexers and registered file watchers (added just go to the queue, changed and deleted cancel already running tasks only for this source) - All files are hashed (content only) to ensure, a change of metadata or tags will not change the hash, and if a file is moved, it will recognize this and update instead of delete and insert The database will contain Tag-Values for every possible value. E.g. File.Location music/album/AC_DC/Back in Black/01 - Hells Bells.mp3 FileTag.Type Artist Tag.Value AC/DC That way I can add a fulltext index on the Tags.Value field containing a searchable value while maintaining the FileTag.Type for recommendations. Let's say I search for `AC/DC`, it will provide an auto-complete for all FileTags.Type values, that show a match + a generic one for searching ALL values: Artist: AC/DC FullText: AC/DC Searching for 2010 will show: Released: 2010 Title: 2010 FullText: 2010 because it contains matches in Releasedate and title. There may be a lot to optimize, but I think my current plan goes pretty well. Let me know what you think about this approach :-) [1]: https://www.navidrome.org/ https://www.navidrome.org/ [2]: https://www.audiobookshelf.org/ https://www.audiobookshelf.org/ [3]: https://beets.io/ https://beets.io/ [4]: https://github.com/sandreas/m4b-tool/ https://github.com/sandreas/m4b-tool/ [5]: https://github.com/sandreas/tone/ https://github.com/sandreas/tone/