2 ms·
This seems like they rewrote Kafka to me. Even moderately sized Kafka clusters can handle the throughput requirement. Can even optimize for performance over du
by xyst 3y ago
This seems like they rewrote Kafka to me.
Even moderately sized Kafka clusters can handle the throughput requirement. Can even optimize for performance over durability.
Some limited query capability with components such as ksqldb.
Maybe offload historical data to blob storage.
Then again, Kafka is kind of complicated to run at these scales. Very easy to fuck up.
- kdazzle 3y agoPlus managed kafka is pretty expensive
- jpgvm 3y agoEh, Kafka isn't easy to fuck up. If anything it's stupidly hard to fuck up assuming you aren't completely incompetent and didn't read up on how it works or the main caveats. I say that because there really isn't much to Kafka in the first place (as long as you aren't including things like Kafka Streams or Kafka Connect etc). Operationally there are some annoying things in OSS Kafka (hot partitions, controller failover slowness pre-KRaft, etc) but it's overall bog simple and easy to work with if you can accept the things it doesn't do (queue-like behavior). I don't love Kafka these days but the fear mongering is a bit much. Sidenote: If you think you want Kafka you should probably check Pulsar first, in most cases you probably want Pulsar or due to changing requirements you would have been better off going Pulsar from the start.