4 ms·
For something like this it would make sense to leverage existing libraries instead of "rolling your own". For the storage engine especially, it's very hard to g
by jiggawatts 4y ago
For something like this it would make sense to leverage existing libraries instead of "rolling your own". For the storage engine especially, it's very hard to get both performance and safety right.
Building on top of Microsoft's FASTER[1] Log and FASTER KV would be a good option for a storage engine, as that is also C# and notably provides higher performance than the in-memory ConcurrentDictionary class in the standard library!
LiteDB uses locks for writers, which means it has virtually no chance of being anywhere near as performant as a lock-free log.
[1] https://github.com/microsoft/FASTER https://github.com/microsoft/FASTER
- neonsunset 4y agoIf you just need a fast key-value map that scales well with threads/cores, might I interest you in non-blocking drop-in replacement for ConcurrentDictionary? https://www.nuget.org/packages/NonBlocking https://www.nuget.org/packages/NonBlocking
- jiggawatts 4y agoLiteDB is a database engine, so it needs persistence. For the link you provided, I wonder if Microsoft would accept a pull request to merge it into the standard library to replace the built-in concurrent dictionary…?
- neonsunset 4y agoSpot on, this was the original intent of the linked implementation but the PR eventually did not progress and got rejected[0]. However, it might not be off the table yet, because scaling per core is dramatically better especially in moderate to write-heavy scenarios which remains the main use case for ConcurrentDictionary. I have been benchmarking it on arm64 lately and it shows different numbers much more favoring non-blocking implementation without any regressions discussed in the PR. [0] https://github.com/dotnet/runtime/pull/50337 https://github.com/dotnet/runtime/pull/50337