We have built a few streaming pipelines at my current place of work, with the standard tech (kafka-confluent, a self-hosted Schema registry, etc) operated for u

We have built a few streaming pipelines at my current place of work, with the standard tech (kafka-confluent, a self-hosted Schema registry, etc) operated for us by SREs. For our use case (messages never expire, consuming logic is complex and evolves every couple of months necessitating a full replay, we’re the only consumer, few 100k messages per year), we’ve decided that it just isn’t worth the trade offs vs storing temporal data in Postgres. The place where streaming seems to be most useful for us is for piping data from one team to any other team. That way, no one needs to quibble over database permissions, we can just hand over the topic name and say “have fun”. For internal workloads, the pain points are the ones where streaming is known to have faults: quick and easy queries on the data in the queue, ergonomics around schema evolution and topic offsets, exactly once delivery with n>1 partitions and shards, Zookeeper crapping out and dropping messages for god knows what reason. These are all problems that Postgres solves for free. I think streaming makes sense between teams, or if your workload is significantly different from ours, but that’s just my two cents. Kafka and friends are for sure useful tools, but they are not the be all end all they’re sometimes made out to be by people who don’t have to deal with them day in and out.