4 ms·

We've had "small-scale" Kafka for a long time. It's an append-only log, and there are a number of ways to implement it, but it's essentially that. The thing th

We've had "small-scale" Kafka for a long time. It's an append-only log, and there are a number of ways to implement it, but it's essentially that. The thing that makes Kafka interesting is the technique of operating from a linux disk write-buffer. That's the trick that makes it fast and scale to huge volumes. But if you don't have the scale, you can stand up a table, or RabbitMQ, or anything that manages append-only ordered log entries. There doesn't need to be a new thing... Kafka was the new thing.

chrsig 3y ago
Yes, there's nothing novel about an append only log. What's missing (or unbeknownst to me) is a library or small server that provides a good general purpose implementation. It's not just a matter of "write to log, done!". Ensuring persistence, keeping track of consumer offsets, transparent compression, waking up consumers on new message availability, support for transactions... It's not just a append only log that's wanted, it's a system for managing append only logs, without the complications like leadership election, replication, partitioning, etc.