3 ms·
Since LinkedIn is using Kafka quite extensively, I'm wondering whether Azures poor support of Apache Kafka is at least partially responsible for this. Azure is
by jabiko 3y ago
Since LinkedIn is using Kafka quite extensively, I'm wondering whether Azures poor support of Apache Kafka is at least partially responsible for this.
Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS.
¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.
- jameshart 3y ago> Azure is developing its own inhouse message broker that they call EventHubs This seems to imply Event hubs is some sort of upcoming beta or something. Event Hubs are one of the oldest parts of Azure IaaS - I think it was launched in 2014?
- jabiko 3y agoI didn't want to imply that its brand new or beta. Just that it is still being actively developed. For example the compacted topic support did get into GA in May 2023.
- TideAd 3y agoLots of Azure stuff is like that. Some big services internally at Microsoft refused to use their Service Fabric product, it couldn't handle the scale & complex deployment needs.
- jiggawatts 3y agoEven at a very small scale I found their Service Fabric platform painful to work with. For comparison, AKS is also quite wonky, but miles ahead of SF.
- isbvhodnvemrwvn 3y agoTheir hosted postgres was pitiful until very recently too.
- PenisBanana 3y agowell, looks like you know what you're talking about.
- karp773 3y agoEmbrace, extend, and extinguish https://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extingu...
- Banditoz 3y agoYou could still run your own Kafka cluster just using their VMs, right? Or is EventHubs cheaper?