3 ms·
> Pony Kafka sends data to Kafka about 5% - 10% slower than librdkafka but reads data from Kafka about 75% slower than librdkafka. What? So what's the point? W
by manigandham 9y ago
> Pony Kafka sends data to Kafka about 5% - 10% slower than librdkafka but reads data from Kafka about 75% slower than librdkafka.
What? So what's the point? Wouldn't it better to just contribute and optimize the existing clients then?
The Kafka/confluent team specifically chose to implement everything in librdkafka (because kafka is client-side logic heavy) and then make thin wrappers for every language so performance, bugs and stability can all be worked on in a single place.
- dmix 9y agoThat was for what I'd imagine is v0.1 quality code, they said there were plenty of optimizations available to make it at least parity with the C client. Secondly, the issue was the system performance between Wallaro and the Kafka client, not necessarily the raw performance of the library.
- hackmanytrades 9y agoYes, that's correct on both counts. Pony Kafka is almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains. And yes, we're concerned about the potential thread pool contention between Pony and librdkafka.
- manigandham 9y agoCopying data in memory between libraries is slower than actually getting the data across the network from the Kafka cluster itself? Seems like a strange bottleneck if so.
- fulafel 9y agoIf they worked on the C client, they then have to work in C, which is bad. (This is one of the things covered in their Tradeoffs section, along with the problems with serialization and the C client's thread pool).