3 ms·

Seems like a lot of what I read about Kafka really makes it sound like using it is quite, well, Kafkaesque Why do so many engineers end up having such a strugg

Seems like a lot of what I read about Kafka really makes it sound like using it is quite, well, Kafkaesque Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. I theorize the following: - Its flexible enough to do things like receive events (messages) and sending downstream events from those received - it can ingest events fast. A well tuned instance is very fast and can handle a lot of volume - it often allows a middleware log point (or other work types) for things happening throughout your whole system Perhaps all of these things (and more) are hard to attain using a different technology

marcinzm 3y ago
The simple approach mentioned in the article gets annoying if you have micro-services that don't share a DB. You could add a shared DB or a nosql DB but then you may as well just add Kafka. Of course the key question then shouldn't be kafka or not-kafka but if you over engineered on micro-services.
quickthrower2 3y ago
- developer tries to get those jobs paying $50k/y more “what kafka, event sourcing and microservices experience do you have”
mrkeen 3y ago
> Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. It's the mental model that's simple: some services write down what happened, other services 'do their own thing' with that information. I can write the core of the system in 2022, with events like 'Joe wants to buy a bike', 'Joe owes us $200', 'Joe paid us $200', 'Send Joe the bike', etc. In 2023 I want to build the book-keeping service, in 2024 I want to build the inventory management system, and in 2025 I want to hook it up to a CRM and see if we can try to sell some bike parts to Joe. Why couldn't I just use REST for that? Because the recipients of the rest calls didn't exist yet. The bad part of Kafka (in my opinion) is how opinionated the consumer logic is (oftentimes by necessity, because of the whole distributed system thing). Sometimes I just wanna ask "what offset are you up to?", but end up in API hell, and am unable to do it.

marcinzm 3y ago

The simple approach mentioned in the article gets annoying if you have micro-services that don't share a DB. You could add a shared DB or a nosql DB but then you may as well just add Kafka. Of course the key question then shouldn't be kafka or not-kafka but if you over engineered on micro-services.

quickthrower2 3y ago

- developer tries to get those jobs paying $50k/y more “what kafka, event sourcing and microservices experience do you have”

mrkeen 3y ago

> Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. It's the mental model that's simple: some services write down what happened, other services 'do their own thing' with that information. I can write the core of the system in 2022, with events like 'Joe wants to buy a bike', 'Joe owes us $200', 'Joe paid us $200', 'Send Joe the bike', etc. In 2023 I want to build the book-keeping service, in 2024 I want to build the inventory management system, and in 2025 I want to hook it up to a CRM and see if we can try to sell some bike parts to Joe. Why couldn't I just use REST for that? Because the recipients of the rest calls didn't exist yet. The bad part of Kafka (in my opinion) is how opinionated the consumer logic is (oftentimes by necessity, because of the whole distributed system thing). Sometimes I just wanna ask "what offset are you up to?", but end up in API hell, and am unable to do it.