3 ms·
So it's a small in-process library for writing and reading append-only logs on disk, that supports multiple threads concurrently reading and writing as a pub/su
by tveita 11y ago
So it's a small in-process library for writing and reading append-only logs on disk, that supports multiple threads concurrently reading and writing as a pub/sub system.
It seems inspired by the Apache Kafka project in that it organizes logs into named topics, but it is not networked, does not split topics into partitions and segments, and does not do housekeeping like deletion of old logs.
It looks pretty new and immature, for instance https://github.com/panyam/KafkaLite/blob/master/src/kltopics.c#L39 https://github.com/panyam/KafkaLite/blob/master/src/kltopics... reads from a potentially uninitialized field and could randomly fail (but will not propagate the error!). Have the tests been run with Valgrind?
Not bad, but it could definitely do with an introduction in the Readme describing its purpose.
- flashgordon 11y agoVery nice pick. Valgrind has been used extensively to test this (valgrind is applied to all tests in the tests package). It looks like when i create my topic array, I zeroed out the memory (with calloc). So the topic is initialised to zero. But this would have resulted in uninitialised read accesses for any new topics that were created beyond the original topic buffer (with initial capacity 32). This has now been fixed (thanks again). I chose to keep networking out of it for now as I felt it was something that could be added by the client (eg by having a websocket listener publish events as it saw fit). Also partitioning was not a high priority now as creation of new topics actually had fairly low overhead. Also deletion and compation of logs is something that was not a high priority right now but definitely something I would like to look at (would love some help :) )