3 ms·

There's a more accurate and detailed explanation of the Kafka protocol here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication https://cwiki.a

There's a more accurate and detailed explanation of the Kafka protocol here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Repl... In step 5 of your example, the new leader does not truncate its log; only the new followers do. Basically, every message that was committed is guaranteed to be in every in-sync replica's log. (Some of these may be after the replica's HW.) But the converse is not true: some replica's logs will contain messages that may not have been committed. So the new leader has to keep its entire log -- which includes all acknowledged messages, plus some unacknowledged ones that become committed as a result of the leadership transition. But the new followers have to discard messages above their HW, because those messages might not be present on the new leader.

majidazimi 9y ago
The doc says: > The replica that registers first becomes the new leader. The new leader chooses its LEO as the new HW. Isn't it a better idea to elect a process who has the largest offset? Basically during leader election, a process should get confirmation of majority of correct processes. If another process observes that it has larger offset in the log, rejects the election and starts another round of election. Eventualy a correct process who has largest offset will become leader. Regarding correctness, this sounds to me as a safer election.
- teraflop 9y ago
  Yeah, I think that would be a reasonable alternate approach. It's not really "safer", though. None of the replicas are guaranteed to have all of the messages that reached the old leader (some of those might never have been replicated anywhere). So clients that didn't receive acknowledgements might need to retry on the new leader, no matter what. Since the clients have to have retry logic anyway, there's no safety issue with discarding some of the messages that weren't yet acknowledged. Your suggestion would try to minimize the number of necessary retries, but at the cost of adding delay and complexity to the leader election process.

majidazimi 9y ago

The doc says: > The replica that registers first becomes the new leader. The new leader chooses its LEO as the new HW. Isn't it a better idea to elect a process who has the largest offset? Basically during leader election, a process should get confirmation of majority of correct processes. If another process observes that it has larger offset in the log, rejects the election and starts another round of election. Eventualy a correct process who has largest offset will become leader. Regarding correctness, this sounds to me as a safer election.

teraflop 9y ago
Yeah, I think that would be a reasonable alternate approach. It's not really "safer", though. None of the replicas are guaranteed to have all of the messages that reached the old leader (some of those might never have been replicated anywhere). So clients that didn't receive acknowledgements might need to retry on the new leader, no matter what. Since the clients have to have retry logic anyway, there's no safety issue with discarding some of the messages that weren't yet acknowledged. Your suggestion would try to minimize the number of necessary retries, but at the cost of adding delay and complexity to the leader election process.