4 ms·

@Aphyr's "Call me maybe: Kafka"[1] blog post is a great reference for what Kafka is and what distributed guarantees it tries to provide. [1]: http://aphyr.com/

by helper 13y ago

@Aphyr's "Call me maybe: Kafka"[1] blog post is a great reference for what Kafka is and what distributed guarantees it tries to provide. [1]: http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka

rdtsc 13y ago
Good point. TL;DR: "Kafka’s replication claimed to be CA, but in the presence of a partition, threw away an arbitrarily large volume of committed writes. It claimed tolerance to F-1 failures, but a single node could cause catastrophe."
- jessaustin 13y ago
  I think I see what he's trying to say, but there's a letter that isn't included in the adjective "CA", and that letter is "P". How could it be otherwise? Indeed, from TFA: "Kafka can do this because LinkedIn’s brokers run in a datacenter, where partitions are rare."
  - arielweisberg 13y ago
    Partitions are != network partitions. Processes on the same node can become partitioned, threads in the same process can become partitioned. Actors within threads can become partitioned from themselves or other actors. In my experience network partitions are not the most common kind of partition although they are well represented, especially on some network topologies. IMO you should never assume that any two things won't be partitioned as part of your expectation of correctness or availability.
    - pfraze 13y ago
      Can you describe a partition of two actors within the same thread? I'd guess this would be in some sort of logic failure state, where either the consuming actor doesn't correctly receive its messages, or the producing actor doesn't correctly transmit.
      - lgierth 13y ago
        It can also happen when one can't keep up with the other and falls behind.
  - Nacraile 13y ago
    You're correct, the system does not advertise partition tolerance, and @Aphyr does indeed confirm that the system is not partition tolerant. Broadly, I think sacrificing P is pretty deeply misguided. If you run a distributed system at any scale for any period of time, you're going to experience network partitions, even within a single DC. (I speak from experience: I work on a distributed system which runs a whole bunch of machines in a whole bunch of datacentres worldwide. We see a reasonable number of non-trivial within-DC network partitions every year.) A much more detailed argument of the above: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/ You really do need to think about what happens to your system when it gets partitioned. This paper has much of interest to say on the topic: http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf
    - derefr 13y ago
      Isn't "a partition-intolerant system" really just another way to say "a non-distributed system"?
      - aaronblohowiak 13y ago
        No, it is just a system that can lose data in the face of a partition.
- Dylan16807 13y ago
  And the reply to that is: "we can tolerate N-1 Kafka node failures, but only N/2-1 Zookeeper failures. This actually is sensible, though, as Kafka node count scales with data size but Zookeeper node count doesn’t. So we would commonly have five Zookeeper replicas but only replication factor 3 within Kafka (even if we have many, many Kafka servers—data in Kafka is partitioned so not all nodes are identical)." (Zookeeper is used to track node membership)
tveita 13y ago
And this is a good explanation of the motivation behind the design, and how it is used in practice: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...
boredandroid 13y ago
I am one of the Kafka committers. I had posted some notes on the Jepsen post which I think may give a more complete picture: http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen http://blog.empathybox.com/post/62279088548/a-few-notes-on-k...

by helper 13y ago

rdtsc 13y ago

Good point. TL;DR: "Kafka’s replication claimed to be CA, but in the presence of a partition, threw away an arbitrarily large volume of committed writes. It claimed tolerance to F-1 failures, but a single node could cause catastrophe."

jessaustin 13y ago
I think I see what he's trying to say, but there's a letter that isn't included in the adjective "CA", and that letter is "P". How could it be otherwise? Indeed, from TFA: "Kafka can do this because LinkedIn’s brokers run in a datacenter, where partitions are rare."
- arielweisberg 13y ago
  Partitions are != network partitions. Processes on the same node can become partitioned, threads in the same process can become partitioned. Actors within threads can become partitioned from themselves or other actors. In my experience network partitions are not the most common kind of partition although they are well represented, especially on some network topologies. IMO you should never assume that any two things won't be partitioned as part of your expectation of correctness or availability.
  - pfraze 13y ago
    Can you describe a partition of two actors within the same thread? I'd guess this would be in some sort of logic failure state, where either the consuming actor doesn't correctly receive its messages, or the producing actor doesn't correctly transmit.
    - lgierth 13y ago
      It can also happen when one can't keep up with the other and falls behind.
- Nacraile 13y ago
  You're correct, the system does not advertise partition tolerance, and @Aphyr does indeed confirm that the system is not partition tolerant. Broadly, I think sacrificing P is pretty deeply misguided. If you run a distributed system at any scale for any period of time, you're going to experience network partitions, even within a single DC. (I speak from experience: I work on a distributed system which runs a whole bunch of machines in a whole bunch of datacentres worldwide. We see a reasonable number of non-trivial within-DC network partitions every year.) A much more detailed argument of the above: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/ You really do need to think about what happens to your system when it gets partitioned. This paper has much of interest to say on the topic: http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf
  - derefr 13y ago
    Isn't "a partition-intolerant system" really just another way to say "a non-distributed system"?
    - aaronblohowiak 13y ago
      No, it is just a system that can lose data in the face of a partition.
Dylan16807 13y ago
And the reply to that is: "we can tolerate N-1 Kafka node failures, but only N/2-1 Zookeeper failures. This actually is sensible, though, as Kafka node count scales with data size but Zookeeper node count doesn’t. So we would commonly have five Zookeeper replicas but only replication factor 3 within Kafka (even if we have many, many Kafka servers—data in Kafka is partitioned so not all nodes are identical)." (Zookeeper is used to track node membership)

tveita 13y ago

And this is a good explanation of the motivation behind the design, and how it is used in practice: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...

boredandroid 13y ago

I am one of the Kafka committers. I had posted some notes on the Jepsen post which I think may give a more complete picture: http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen http://blog.empathybox.com/post/62279088548/a-few-notes-on-k...