4 ms·

My team at Parse.ly also did a benchmark comparing pykafka (pure Python) to pykafka with the librdkafka C extension enabled. That C module is clearly a huge win

My team at Parse.ly also did a benchmark comparing pykafka (pure Python) to pykafka with the librdkafka C extension enabled. That C module is clearly a huge win for Kafka consumer/producer performance on Python and other dynamic languages. http://blog.parsely.com/post/3886/pykafka-now/ http://blog.parsely.com/post/3886/pykafka-now/ Unfortunately, as the OP illustrates, there are now 2 widely-used Python + Kafka drivers (pykafka and kafka-python), and as of recently, a third, confluent-kafka-python, which is a thin wrapper over librdkafka. The reason there's all this fragmentation is because Kafka was quite the moving target for non-JVM languages for the past three years. We have used it in production since Kafka 0.7, so we've had to live through it all blow-by-blow. I'm hoping that with Kafka 0.10 recently released, we can finally unify the community around a single driver (somehow).

dpkp 10y ago
I enjoyed your blog post, but I don't think this is a fair characterization: kafka-python is not "mostly 0.9+ focused..." kafka-python is the only driver that is both forward and backwards compatible w/ kafka 0.8 through 0.10. As I'm sure you remember, kafka-python was the original 0.8 driver, written to support the 0.8 protocol b/c Samsa (pykafka's previous incarnation) was only supporting 0.7 and did not have any plans to upgrade.
pixelmonkey 10y ago
@dkfp Apologies for that, I did not mean to mis-characterize. PyKafka also goes from 0.8 => 0.10. I had assumed kafka-python recently switched to be 0.9-only due to all the changes related to consumer groups.
- dpkp 10y ago
  No apology required. Though note that pykafka requires >=0.8.2 , and is only forwards compatible w/ newer brokers. This means that pykafka implements the 0.8.2 feature set. Newer brokers support that feature set, but you are not taking advantage of 0.9 or 0.10 features if you connect to them. kafka-python, on the other hand is both forwards and backwards compatible. It supports all feature sets: from no offsets in 0.8, to zk offests in 0.8.1, to kafka offsets in 0.8.2, to group management in 0.9, to message timestamps and relative-offset compressed messages in 0.10. The feature set to use is chosen based on the broker version we're connected to. As far as I know, no other client supports this approach -- not python, not java, etc. [Though KIP-35 should open this up to other clients for backwards compatibility starting at 0.10]
  - emmett9001 10y ago
    Pykafka does currently have support for 0.9 group management, and we intend to add support for message timestamps and the other new 0.10 features. We're not, however, detecting the broker version and turning on features on that basis. Instead we prefer to let the user explicitly enable the features they're interested in using.
jdennison 10y ago
There is obvious value in a having a pure python implementation of a kafka client. Many deployments don't want C extensions or want to use pypy. However, as python's scipy stack has shown, the right python api wrapping C code can have a vibrant community and the speed to boot.

dpkp 10y ago

I enjoyed your blog post, but I don't think this is a fair characterization: kafka-python is not "mostly 0.9+ focused..." kafka-python is the only driver that is both forward and backwards compatible w/ kafka 0.8 through 0.10. As I'm sure you remember, kafka-python was the original 0.8 driver, written to support the 0.8 protocol b/c Samsa (pykafka's previous incarnation) was only supporting 0.7 and did not have any plans to upgrade.

pixelmonkey 10y ago

@dkfp Apologies for that, I did not mean to mis-characterize. PyKafka also goes from 0.8 => 0.10. I had assumed kafka-python recently switched to be 0.9-only due to all the changes related to consumer groups.

dpkp 10y ago
No apology required. Though note that pykafka requires >=0.8.2 , and is only forwards compatible w/ newer brokers. This means that pykafka implements the 0.8.2 feature set. Newer brokers support that feature set, but you are not taking advantage of 0.9 or 0.10 features if you connect to them. kafka-python, on the other hand is both forwards and backwards compatible. It supports all feature sets: from no offsets in 0.8, to zk offests in 0.8.1, to kafka offsets in 0.8.2, to group management in 0.9, to message timestamps and relative-offset compressed messages in 0.10. The feature set to use is chosen based on the broker version we're connected to. As far as I know, no other client supports this approach -- not python, not java, etc. [Though KIP-35 should open this up to other clients for backwards compatibility starting at 0.10]
- emmett9001 10y ago
  Pykafka does currently have support for 0.9 group management, and we intend to add support for message timestamps and the other new 0.10 features. We're not, however, detecting the broker version and turning on features on that basis. Instead we prefer to let the user explicitly enable the features they're interested in using.

jdennison 10y ago

There is obvious value in a having a pure python implementation of a kafka client. Many deployments don't want C extensions or want to use pypy. However, as python's scipy stack has shown, the right python api wrapping C code can have a vibrant community and the speed to boot.