4 ms·

In a me too way, I also created something similar - https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb It started on top of bbolt, but over

by Ingon 3y ago

In a me too way, I also created something similar - https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb It started on top of bbolt, but over time moved a completely custom implementation. The main reason for this was performance, but I also wanted to run a lot of queues, so each one had to be as light as possible. On top it even provides some indexing capabilities, which (when enabled) allow for quick find by key and time.

Xeoncross 3y ago
Interesting, the performance of https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss? I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).
- Ingon 3y ago
  First off, looking at goqlite benchmarks, we are measuring different things - goqlite is measuring a complete lifecycle of a message, which includes sending, receiving and deleting. The klevdb benchmarks are usually exercising one (at most 2) of these. In klevdb you can control the fsync yourself - using AutoSync you can ask it to perform fsycn after each Publish, which currently is best effort put on disk. Of course in this case performance suffer dramatically (as you would expected, esp if you don't batch). You can also not use AutoSync and control it yourself (klevdb exposes Sync), deciding what kind of message loss you are prepared to deal with. The benchmarks you see run without AutoSync, to highlight what kind of raw throughput klevdb is capable of. And yeah, klevdb is basically an append-only log. Of course, you can delete messages from it, but this is done in parallel, where the segment is being copied without the messages you want to delete, then replaced with a single mv system call. Rewriting the head segment (e.g. the one you are writing to) is not always possible (since there could be messages the rewriter haven't seen), but will eventually succeed (once the head segment is no longer a head). Hope this answers your questions, and if not, happy to expand more. I've been contemplating on implementing transactions on top of klevdb, but haven't quite figured out if I need them or not. Edit: spelling and grammar

by Ingon 3y ago

Xeoncross 3y ago

Interesting, the performance of https://github.com/klev-dev/klevdb https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss? I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).

Ingon 3y ago
First off, looking at goqlite benchmarks, we are measuring different things - goqlite is measuring a complete lifecycle of a message, which includes sending, receiving and deleting. The klevdb benchmarks are usually exercising one (at most 2) of these. In klevdb you can control the fsync yourself - using AutoSync you can ask it to perform fsycn after each Publish, which currently is best effort put on disk. Of course in this case performance suffer dramatically (as you would expected, esp if you don't batch). You can also not use AutoSync and control it yourself (klevdb exposes Sync), deciding what kind of message loss you are prepared to deal with. The benchmarks you see run without AutoSync, to highlight what kind of raw throughput klevdb is capable of. And yeah, klevdb is basically an append-only log. Of course, you can delete messages from it, but this is done in parallel, where the segment is being copied without the messages you want to delete, then replaced with a single mv system call. Rewriting the head segment (e.g. the one you are writing to) is not always possible (since there could be messages the rewriter haven't seen), but will eventually succeed (once the head segment is no longer a head). Hope this answers your questions, and if not, happy to expand more. I've been contemplating on implementing transactions on top of klevdb, but haven't quite figured out if I need them or not. Edit: spelling and grammar