3 ms·

because the only sane write mode in kafka is “ack=-1” which mean wait for confirmation from all in sync replica. The more replica you have the worse is your wri

by skyde 6y ago

because the only sane write mode in kafka is “ack=-1” which mean wait for confirmation from all in sync replica. The more replica you have the worse is your write latency. In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away.

EdwardDiego 6y ago
We use acks=1 just fine. At least once is pretty okay for a lot of use cases. > In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away. I agree, all good points about Pulsar using BK, although worth noting the autorecovery daemon has to be explicitly enabled. But I'll mention Cruise Control again - it solves a all of the issues you've described without manual intervention. https://github.com/linkedin/cruise-control https://github.com/linkedin/cruise-control
- skyde 6y ago
  Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle. Cruise control try to patch around the bad design of kafka but there is only little it can do. Any broker in ISR going down still cause big latency spike , and if broker was leader for several partition all those partition will be down until the single cluster controller finish doing leader switch sequentially for all those partitions. Typically on cluster with many partition of topic, each broker would be leader for ~1000 partition which mean it will take up to 15 minutes for leader switch to complete for all partitions.
  - EdwardDiego 6y ago
    Yep, agree that Pulsar is far better for dealing with large amounts of topics than Kafka. Doesn't mean that Kafka is broken, it just means it has limitations.
- skyde 6y ago
  if your use case is ok with ack=1 then kafka is perfect for you. because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. if you care more about maximum throughput and efficiency than about data integrity and availability then kafka is a good choice.
  - EdwardDiego 6y ago
    > because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. Nope - the leader only acks when the write has been committed to disk. It's called at-least-once because it could fail between committing to disk and acking, and then the producer would try again. You're thinking of acks=0, at-most-once.
    - morelisp 6y ago
      No, the parent is correct. acks=1 can still lose data as the broker can lose topic leadership after write but before replication. Roughly you can think of it as: - acks=0 - can tolerate only a small (but most common!) subset of failures; better to keep going with the next datum than fail/retry if anything is wrong, stale data might as well be no data or you only need 99% of it anyway; very common for web traffic analysis, hardware sensor data, etc; - acks=1 - can tolerate client, client/broker, or single broker failure as long as client can retry; nice for e.g. API uploads where the client has more work to get onto but also wants to provide some reasonable feedback to the caller; a good default; - acks=-1 - can tolerate client, client/broker, broker/broker, or arbitrary broker failure as long as client can retry; good for events which should absolutely not be lost and small enough to keep locally e.g. "batch job finished", "IDS alerted". "At least once" and "at most once" are more often used to describe consumer commit behaviors than producer.

by skyde 6y ago

EdwardDiego 6y ago

We use acks=1 just fine. At least once is pretty okay for a lot of use cases. > In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away. I agree, all good points about Pulsar using BK, although worth noting the autorecovery daemon has to be explicitly enabled. But I'll mention Cruise Control again - it solves a all of the issues you've described without manual intervention. https://github.com/linkedin/cruise-control https://github.com/linkedin/cruise-control

skyde 6y ago
Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle. Cruise control try to patch around the bad design of kafka but there is only little it can do. Any broker in ISR going down still cause big latency spike , and if broker was leader for several partition all those partition will be down until the single cluster controller finish doing leader switch sequentially for all those partitions. Typically on cluster with many partition of topic, each broker would be leader for ~1000 partition which mean it will take up to 15 minutes for leader switch to complete for all partitions.
- EdwardDiego 6y ago
  Yep, agree that Pulsar is far better for dealing with large amounts of topics than Kafka. Doesn't mean that Kafka is broken, it just means it has limitations.
skyde 6y ago
if your use case is ok with ack=1 then kafka is perfect for you. because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. if you care more about maximum throughput and efficiency than about data integrity and availability then kafka is a good choice.
- EdwardDiego 6y ago
  > because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. Nope - the leader only acks when the write has been committed to disk. It's called at-least-once because it could fail between committing to disk and acking, and then the producer would try again. You're thinking of acks=0, at-most-once.
  - morelisp 6y ago
    No, the parent is correct. acks=1 can still lose data as the broker can lose topic leadership after write but before replication. Roughly you can think of it as: - acks=0 - can tolerate only a small (but most common!) subset of failures; better to keep going with the next datum than fail/retry if anything is wrong, stale data might as well be no data or you only need 99% of it anyway; very common for web traffic analysis, hardware sensor data, etc; - acks=1 - can tolerate client, client/broker, or single broker failure as long as client can retry; nice for e.g. API uploads where the client has more work to get onto but also wants to provide some reasonable feedback to the caller; a good default; - acks=-1 - can tolerate client, client/broker, broker/broker, or arbitrary broker failure as long as client can retry; good for events which should absolutely not be lost and small enough to keep locally e.g. "batch job finished", "IDS alerted". "At least once" and "at most once" are more often used to describe consumer commit behaviors than producer.