4 ms·

if your use case is ok with ack=1 then kafka is perfect for you. because this mean you don’t care about losing some of the messages for which the broker confir

by skyde 6y ago

if your use case is ok with ack=1 then kafka is perfect for you. because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. if you care more about maximum throughput and efficiency than about data integrity and availability then kafka is a good choice.

EdwardDiego 6y ago
> because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. Nope - the leader only acks when the write has been committed to disk. It's called at-least-once because it could fail between committing to disk and acking, and then the producer would try again. You're thinking of acks=0, at-most-once.
- morelisp 6y ago
  No, the parent is correct. acks=1 can still lose data as the broker can lose topic leadership after write but before replication. Roughly you can think of it as: - acks=0 - can tolerate only a small (but most common!) subset of failures; better to keep going with the next datum than fail/retry if anything is wrong, stale data might as well be no data or you only need 99% of it anyway; very common for web traffic analysis, hardware sensor data, etc; - acks=1 - can tolerate client, client/broker, or single broker failure as long as client can retry; nice for e.g. API uploads where the client has more work to get onto but also wants to provide some reasonable feedback to the caller; a good default; - acks=-1 - can tolerate client, client/broker, broker/broker, or arbitrary broker failure as long as client can retry; good for events which should absolutely not be lost and small enough to keep locally e.g. "batch job finished", "IDS alerted". "At least once" and "at most once" are more often used to describe consumer commit behaviors than producer.

by skyde 6y ago

EdwardDiego 6y ago

> because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. Nope - the leader only acks when the write has been committed to disk. It's called at-least-once because it could fail between committing to disk and acking, and then the producer would try again. You're thinking of acks=0, at-most-once.

morelisp 6y ago
No, the parent is correct. acks=1 can still lose data as the broker can lose topic leadership after write but before replication. Roughly you can think of it as: - acks=0 - can tolerate only a small (but most common!) subset of failures; better to keep going with the next datum than fail/retry if anything is wrong, stale data might as well be no data or you only need 99% of it anyway; very common for web traffic analysis, hardware sensor data, etc; - acks=1 - can tolerate client, client/broker, or single broker failure as long as client can retry; nice for e.g. API uploads where the client has more work to get onto but also wants to provide some reasonable feedback to the caller; a good default; - acks=-1 - can tolerate client, client/broker, broker/broker, or arbitrary broker failure as long as client can retry; good for events which should absolutely not be lost and small enough to keep locally e.g. "batch job finished", "IDS alerted". "At least once" and "at most once" are more often used to describe consumer commit behaviors than producer.