3 ms·

The main aspect for me after 5 years of running several Kafka cluster to run production critical system at Microsoft is that it’s very fragile because of the ba

by skyde 6y ago

The main aspect for me after 5 years of running several Kafka cluster to run production critical system at Microsoft is that it’s very fragile because of the bad design. distributed system are hard to design correctly. When using HDFS you can create a single 1TB file if the cluster have enough capacity. And with (hdfs hbase cassandra ..) if 5 server permanently fail in the span of 24 hours you don’t need manual intervention and your data is not lost. In Kafka your partition data need to fit entirely on a single server and you might still run into issue if that server also hosting other partition. And with Kafka if one by one the 3 server manually assigned as replica to one partition fail before an engineer is wake up in middle of the night and have time to fix it manually you just lost all your data.

LgWoodenBadger 6y ago
Do you have examples of other systems that won’t lose all your data if all its replicas fail? This seems unavoidable in any scenario.
- skyde 6y ago
  Assuming replication factor of 3 i’m not talking about 3 machine going down at exact same time, but in sequence. like 5 minutes apart. All other distributed system handle this with no issue. Ex: Cassandra, Hbase/hdfs, CockroachDB
EdwardDiego 6y ago
If you're losing machines like that, why not set replicas higher? I'm assuming you're running a rather large cluster if 3 machines failing is fairly routine. Also, did you look into Cruise Control at all? But I agree that having to fit any entire replica of a partition on one machine is a weakness of Kafka, and something that Pulsar improves on by using Bookkeeper which only needs to be able to store a segment of a ledger - BK's autorecovery daemon is nice too.
- skyde 6y ago
  because the only sane write mode in kafka is “ack=-1” which mean wait for confirmation from all in sync replica. The more replica you have the worse is your write latency. In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away.
  - EdwardDiego 6y ago
    We use acks=1 just fine. At least once is pretty okay for a lot of use cases. > In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away. I agree, all good points about Pulsar using BK, although worth noting the autorecovery daemon has to be explicitly enabled. But I'll mention Cruise Control again - it solves a all of the issues you've described without manual intervention. https://github.com/linkedin/cruise-control https://github.com/linkedin/cruise-control
    - skyde 6y ago
      Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle. Cruise control try to patch around the bad design of kafka but there is only little it can do. Any broker in ISR going down still cause big latency spike , and if broker was leader for several partition all those partition will be down until the single cluster controller finish doing leader switch sequentially for all those partitions. Typically on cluster with many partition of topic, each broker would be leader for ~1000 partition which mean it will take up to 15 minutes for leader switch to complete for all partitions.
      - EdwardDiego 6y ago

by skyde 6y ago

LgWoodenBadger 6y ago

Do you have examples of other systems that won’t lose all your data if all its replicas fail? This seems unavoidable in any scenario.

skyde 6y ago
Assuming replication factor of 3 i’m not talking about 3 machine going down at exact same time, but in sequence. like 5 minutes apart. All other distributed system handle this with no issue. Ex: Cassandra, Hbase/hdfs, CockroachDB

EdwardDiego 6y ago

If you're losing machines like that, why not set replicas higher? I'm assuming you're running a rather large cluster if 3 machines failing is fairly routine. Also, did you look into Cruise Control at all? But I agree that having to fit any entire replica of a partition on one machine is a weakness of Kafka, and something that Pulsar improves on by using Bookkeeper which only needs to be able to store a segment of a ledger - BK's autorecovery daemon is nice too.

skyde 6y ago
because the only sane write mode in kafka is “ack=-1” which mean wait for confirmation from all in sync replica. The more replica you have the worse is your write latency. In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away.
- EdwardDiego 6y ago
  We use acks=1 just fine. At least once is pretty okay for a lot of use cases. > In the case of pulsar ack quorum can be much smaller than replication factor so you don’t have to wait for slow replica. Also no number of node going down would cause new write to fail. And under replicated segment will fixed automatically using remaining healthy machines right away. I agree, all good points about Pulsar using BK, although worth noting the autorecovery daemon has to be explicitly enabled. But I'll mention Cruise Control again - it solves a all of the issues you've described without manual intervention. https://github.com/linkedin/cruise-control https://github.com/linkedin/cruise-control
  - skyde 6y ago
    Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle. Cruise control try to patch around the bad design of kafka but there is only little it can do. Any broker in ISR going down still cause big latency spike , and if broker was leader for several partition all those partition will be down until the single cluster controller finish doing leader switch sequentially for all those partitions. Typically on cluster with many partition of topic, each broker would be leader for ~1000 partition which mean it will take up to 15 minutes for leader switch to complete for all partitions.
    - EdwardDiego 6y ago