4 ms·

Not sure I agree. It seems as good a way as any to decouple systems, asynchronously exchange messages between services, get them into durable storage, exactly

Not sure I agree. It seems as good a way as any to decouple systems, asynchronously exchange messages between services, get them into durable storage, exactly once processing, replay semantics etc. I think it should be on the table at least whenever we have two services needing to exchange data. Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations. What alternatives would you be looking at?

weego 5y ago
It is as good as any if you have no financial constraints and no technical overhead and time constraints, but everyone does. Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.
- hansbo 5y ago
  What would you say is its baggage?
  - berkes 5y ago
    For me, the baggage is mostly the complexity of the service. With that comes monitoring, maintenance, tuning, debugging and troubleshooting. Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close). Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
    - piyh 5y ago
      I second Kafka having massive operational overhead. It's a burden and is killing any support for it within our org.
berkes 5y ago
Some alternatives are: * Just keep your architecture a monolith. You'll do fine the majority of the cases. * Event-sourcing doesn't require Kafka clusters. Nor do event-driven setups. You don't need complex tooling to pass around strings/json-blurps. An S3 bucket or a Postgresql database storing "Events-as-json" is often fine. * Postgres can do most of what you need (except for the "webscale" clustering etc)[0] in practice already. * Redis[1] My main point is that while Kafka is a fantastic tool, you don't need that tool to achieve what you want in many cases. > It seems as good a way as any to decouple systems IMO relying on a tool to achieve a good software design, rather than design-patterns, is a recipe for trouble. If anything, because it locks you in (do you suddenly get a tightly coupled system if you remove Kafka?) or because its details force you into directions that don't naturally fit your domain or problem. -- [0] https://spin.atomicobject.com/2021/02/04/redis-postgresql/ https://spin.atomicobject.com/2021/02/04/redis-postgresql/ [1] https://redis.com/redis-best-practices/communication-patterns/event-queue/ https://redis.com/redis-best-practices/communication-pattern... etc.
LgWoodenBadger 5y ago
I’ve seen comments like the gp’s often enough that it strikes me as a form of gatekeeping. MY problems are so special that my use of Kafka was perfect, but YOURS are trivial and you shouldn’t even consider Kafka.
- robertlagrant 5y ago
  > it strikes me as a form of gatekeeping I consider this a form of gatekeeping of advice on using Kafka.

weego 5y ago

It is as good as any if you have no financial constraints and no technical overhead and time constraints, but everyone does. Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.

hansbo 5y ago
What would you say is its baggage?
- berkes 5y ago
  For me, the baggage is mostly the complexity of the service. With that comes monitoring, maintenance, tuning, debugging and troubleshooting. Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close). Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
  - piyh 5y ago
    I second Kafka having massive operational overhead. It's a burden and is killing any support for it within our org.

berkes 5y ago

Some alternatives are: * Just keep your architecture a monolith. You'll do fine the majority of the cases. * Event-sourcing doesn't require Kafka clusters. Nor do event-driven setups. You don't need complex tooling to pass around strings/json-blurps. An S3 bucket or a Postgresql database storing "Events-as-json" is often fine. * Postgres can do most of what you need (except for the "webscale" clustering etc)[0] in practice already. * Redis[1] My main point is that while Kafka is a fantastic tool, you don't need that tool to achieve what you want in many cases. > It seems as good a way as any to decouple systems IMO relying on a tool to achieve a good software design, rather than design-patterns, is a recipe for trouble. If anything, because it locks you in (do you suddenly get a tightly coupled system if you remove Kafka?) or because its details force you into directions that don't naturally fit your domain or problem. -- [0] https://spin.atomicobject.com/2021/02/04/redis-postgresql/ https://spin.atomicobject.com/2021/02/04/redis-postgresql/ [1] https://redis.com/redis-best-practices/communication-patterns/event-queue/ https://redis.com/redis-best-practices/communication-pattern... etc.

LgWoodenBadger 5y ago

I’ve seen comments like the gp’s often enough that it strikes me as a form of gatekeeping. MY problems are so special that my use of Kafka was perfect, but YOURS are trivial and you shouldn’t even consider Kafka.

robertlagrant 5y ago
> it strikes me as a form of gatekeeping I consider this a form of gatekeeping of advice on using Kafka.