2 ms·
It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config
by pram 5y ago
It can become the opposite of sensible, very easily. I've had devs demand to use Kafka for storing 100mb+ blobs (which of course required a custom broker config + cluster)
- geodel 5y agoIndeed. The idea that somehow Kafka is beyond reproach seems amusing to me.
- the_af 5y agoThankfully nobody here said that Kafka is beyond reproach. I just said the idea behind Kafka, the distributed log, makes a lot of sense.
- the_af 5y agoAnything can become the opposite of sensible, so that's not saying much. I'm saying there are logical, sensible ideas behind things such as Kafka -- "you could have invented the distributed log" -- and that it's not snake oil. Of course it can be misused, not everything calls for it, and it comes with tradeoffs and pitfalls, but which tool doesn't?
- pram 5y agoThere is a logical idea behind it, of course! I didn't say it was snake oil anywhere. Kafka was even pretty novel 7 years ago when I started deploying it (although 0.9 was an enormous dumpster fire) Today you also have Pulsar, Redpanda, Kinesis, PubSub, and NATS off the top of my head. You can do the same streaming patterns with all of them, with the exception of log compaction probably. It's an architecture so amazing that the world needs 6 redundant implementations. I think my OP implied something regarding that.
- the_af 5y agoI agree with you that the slightly different implementations of the same idea are confusing and slightly embarrassing. No argument from me there!
- agallego 5y agofor the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets