4 ms·

What issues did you run into? From a technology perspective it's been rock solid for years in my experience. Where issues crept in it was always due to people

What issues did you run into? From a technology perspective it's been rock solid for years in my experience. Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the effects of partitioning, resetting offsets on a live topic, aggressive retention policies etc.

taywrobel 4y ago
One issue I’ve encountered is over-partitioning to handle a spike in traffic. I.e. an event occurs which causes an order of magnitude more messages than usual to be produced for a couple of hours, and because ingest and processing flows are out of whack, a backlog forms. Management wants things back in sync ASAP, and so green lights increasing the partition count on the topic, usually doubling it. In an event driven architecture that is fairly well tuned for normal traffic this can have the same downstream effect, and those topics up their partition counts as well in response. Once anomalous traffic subsides, teams go to turn down the now over-partitioned topics only to learn that that was a one way operation and now they’re stuck with that many partitions, and the associated cost overhead. Also if I see another team try to implement “retries” or delayed processing on messages by doing some weird multi-topic trickery I’m going to lose my mind. Kafka is a message queue, not a job queue, and not nearly enough engineers seem to grok that.
TheSoftwareGuy 4y ago
Do you know of any places to learn those things? Kafka seems pretty interesting to me
- ryanjshaw 4y ago
  - Confluent blog, particularly anything by Martin Kleppmann: https://www.confluent.io/blog/author/martin-kleppmann/ https://www.confluent.io/blog/author/martin-kleppmann/ - "I Heart Logs" by Jay Krepps (free) - "Kafka, The Definitive Guide" (also free)

taywrobel 4y ago

One issue I’ve encountered is over-partitioning to handle a spike in traffic. I.e. an event occurs which causes an order of magnitude more messages than usual to be produced for a couple of hours, and because ingest and processing flows are out of whack, a backlog forms. Management wants things back in sync ASAP, and so green lights increasing the partition count on the topic, usually doubling it. In an event driven architecture that is fairly well tuned for normal traffic this can have the same downstream effect, and those topics up their partition counts as well in response. Once anomalous traffic subsides, teams go to turn down the now over-partitioned topics only to learn that that was a one way operation and now they’re stuck with that many partitions, and the associated cost overhead. Also if I see another team try to implement “retries” or delayed processing on messages by doing some weird multi-topic trickery I’m going to lose my mind. Kafka is a message queue, not a job queue, and not nearly enough engineers seem to grok that.

TheSoftwareGuy 4y ago

Do you know of any places to learn those things? Kafka seems pretty interesting to me

ryanjshaw 4y ago
- Confluent blog, particularly anything by Martin Kleppmann: https://www.confluent.io/blog/author/martin-kleppmann/ https://www.confluent.io/blog/author/martin-kleppmann/ - "I Heart Logs" by Jay Krepps (free) - "Kafka, The Definitive Guide" (also free)