5 ms·

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by

by ovis 6y ago

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

rad_gruchalski 6y ago
Re 2, have you ever tried using Kafka with a non-JVM client?
- EdwardDiego 6y ago
  Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
  - rad_gruchalski 6y ago
    I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.
    - EdwardDiego 6y ago
      Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions. IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going. That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients. Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.
- mrkeen 6y ago
  kafkacat is excellent. https://github.com/edenhill/kafkacat https://github.com/edenhill/kafkacat
  - rad_gruchalski 6y ago
    Definitely, if all one needs is command line.
skyde 6y ago
for #1 if you need message order to be maintained you cannot use more than 1 partition. for #2 99% of the production issue we had where caused by bugs in librdkafka not talking to the right brokers. Also it mean that the IP for the broker need to all be public IP and the writing throughput and latency is limited by the capacity of the machine that is the master for the partition you are trying to write. If the machine hosting you partition become overloaded you have to switch the master for that partition to another machine unlike pulsar this is not done automatically and also if you replication factor is 3 your choice are limited to 2 other machine if you don't want to copy the whole partition to a new machine which would take hours.

by ovis 6y ago

rad_gruchalski 6y ago

Re 2, have you ever tried using Kafka with a non-JVM client?

EdwardDiego 6y ago
Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
- rad_gruchalski 6y ago
  I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.
  - EdwardDiego 6y ago
    Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions. IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going. That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients. Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.
mrkeen 6y ago
kafkacat is excellent. https://github.com/edenhill/kafkacat https://github.com/edenhill/kafkacat
- rad_gruchalski 6y ago
  Definitely, if all one needs is command line.

skyde 6y ago

for #1 if you need message order to be maintained you cannot use more than 1 partition. for #2 99% of the production issue we had where caused by bugs in librdkafka not talking to the right brokers. Also it mean that the IP for the broker need to all be public IP and the writing throughput and latency is limited by the capacity of the machine that is the master for the partition you are trying to write. If the machine hosting you partition become overloaded you have to switch the master for that partition to another machine unlike pulsar this is not done automatically and also if you replication factor is 3 your choice are limited to 2 other machine if you don't want to copy the whole partition to a new machine which would take hours.