3 ms·
(disclaimer: Akka team here). The post, somewhat surprisingly, omits to mention Akka Streams (which is a perfect fit, since, as the post mentions 'The data cam
by ktoso 10y ago
(disclaimer: Akka team here).
The post, somewhat surprisingly, omits to mention Akka Streams (which is a perfect fit, since, as the post mentions 'The data came through a stream [...]'), and Reactive Kafka (which is an official Akka project https://github.com/akka/reactive-kafka https://github.com/akka/reactive-kafka ) which solve this exact use case.
These projects/modules have been around since 2014, and we've been presenting / sharing information about them a lot since then. Perhaps this post can be interpreted that we need to put even more effort into the discoverability of them (in fact, we are right now reworking documentation for this and more). Using Akka Streams, a complete re-implementation of the example use-case mentioned in the post would look like this:
`Consumer.plainSource(kafkaConsumerSettings, kafkaSubscription).groupedWithin(maxItemsInBatch, 100.milliseconds).runForeach(batch => persist(batch))`
Too bad the team missed these... I would have loved to point the team to the right (existing) abstraction/library in one of our many communities (github, gitter chat and the mailing lists - we're pretty nice people, come around some time), rather than posting like this. What we've learnt here though is that we need to work even harder on the discoverability of those libraries - and indeed it is one of the things we focus on nowadays (with docs re-designs, cross links, better search and more).
Anyway, just wanted to let you all know what Akka has in store for such use cases, Streaming is definitely a first class citizen in the toolkit.