6 ms·
Guys firstly thank you very much for the exceptional feedback. And I apologise for the missing details. I found that as I was adding more to this the more I
by flashgordon 11y ago
Guys firstly thank you very much for the exceptional feedback. And I apologise for the missing details. I found that as I was adding more to this the more I felt I had to add (fear of publishing really). So I wanted to push one out so I don't end up waiting for the perfect thing for ever.
In short the purpose of this library was to allow any application to embed an event broker with persistence so that it could consume events at its own pace. The purpose of this library is NOT to replace or compete with GCD or java.util.conc (which provide excellent primitives to do in memory queuing and processing of events).
The main use case that we wanted this for was message synchronisation. Consider a mobile messaging application that allows users (on mobile clients) to send and receive messages as well as do other CRUD ops on messages . To synchronise these events across all clients (say users in a channel) we would have to keep a connection (eg websockets) open to the server that would receive publish events to the client or poll for change events periodically by the client (of course the server would have to maintain its own event log - but a Kafka event sprayer that publishes to websockets is easily done).
In either case, when these events are received by the client they need to be processed as soon as they are received or the client would have to refetch them (harder to do on a non-polling situation as the state of what events have been consumed is expensive to record and maintain). Problem with having to process these events as soon as they are received is that there is no way to prioritise the events and there is no way to defer processing to a time convenient to the client. So with this library we have an append only log of events onto which we persist the events as we receive them (and because we can have multiple topics for each KafkaLite - KL - context) it allows us to impose prioritisation on which events are to be processed when.
I have not built in replication replication as it can be built on top as and when required. Also sharing of messages on a particular topic across processes while doable was not an immediate priority (would love to hear other use cases for this) so was eschewed. One other thing I would definitely like to do is provide a few sample producers that are platform specific (eg websockets based publishers, polling based publishers) but these seemed not generic enough (at least as of now) so I have not had a chance to investigate these further.
I would also like to apologise for any confusion with the name (especially to the gods of Apache Kafka!). As I had mentioned, this was only inspired by Kafka's original design (ie a stateless, fast and append only log). This library is written in C so as to be portable across any platform and not having to incur the cost bundling and invoking a JVM (clearly there is a lot to be done for this to be useable on the server side) - Hence the "Lite".
Again I really appreciate the feedback. Even though you see it when you read others content, with your own content you start with the bias of clarity as I did. Even more so if any of you would like help in using it in your own applications I would love to have a chat as to what features we could add to improve the robustness and usefulness.
- flashgordon 11y agoIl definitely add this to the README :)
- toomim 11y ago"I have not built in replication replication" Lol, you replicated the word.
- flashgordon 11y agoHaha... Just to show I take replication seriously :)
- rakoo 11y ago> So with this library we have an append only log of events onto which we persist the events as we receive them You could also use any member of the CouchDB family, such as PouchDB for the browser (even as a lightweight desktop solution), or any member for the mobile [0] [1] [2] [0] https://github.com/couchbase/couchbase-lite-net https://github.com/couchbase/couchbase-lite-net [1] https://github.com/couchbase/couchbase-lite-android https://github.com/couchbase/couchbase-lite-android [2] https://github.com/couchbase/couchbase-lite-ios https://github.com/couchbase/couchbase-lite-ios What could be really interesting is another member as a library; I don't think there exists any, but kafkalite could be the beginning of something...
- flashgordon 11y agoHey rakoo thanks for sharing that. I actually came across this earlier. (Also RocksDB /LevelDB are amazing as embedded key-value stores). My starting point was to see what a pure log would give you and let the application worry about what it did with the events (eg in the messaging case - update the messages in the respective channels with the new content - the channel is just a table in sqlite!) - essentially a way to decouple the processing of events from their transport and persistence.