3 ms·

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be inte

For Kafka alternative written in C++ there's Redpanda [1],[2]. Redpanda claim of better performance but benchmarks showed no clear winner [3]. It will be interesting to test them together on the performance benchmarks. I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus). It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4]. [1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments): https://news.ycombinator.com/item?id=25075739 https://news.ycombinator.com/item?id=25075739 [2] Redpanda website: https://www.redpanda.com/ https://www.redpanda.com/ [3] Kafka vs. Redpanda performance – do the claims add up? (141 comments): https://news.ycombinator.com/item?id=35949771 https://news.ycombinator.com/item?id=35949771 [4] What If We Could Rebuild Kafka from Scratch? (220 comments): https://news.ycombinator.com/item?id=43790420 https://news.ycombinator.com/item?id=43790420

nubskr 11mo ago
In the current benchmarks, I only have Kafka and rocksdb wal, will surely try to add redpanda there as well, curious how walrus would hold up against seastar based systems.
- chaotic-good 11mo ago
  I don't see any mentions of p99 latency in the benchmark results. Pushing gigabytes per second is not that difficult on modern hardware. Doing so with reasonable latency is what's challenging. Also, instead of using custom benchmarks it's better to just use the OMB (open-messaging benchmark).
EdwardDiego 11mo ago
> It's the very architecture of Kafka itself due to the notorious head of line problem Except a consumer can discard an unprocessable record? I'm not certain I understand how HOL applies to Kafka, but keen to learn more :)
- thinkharderdev 11mo ago
  > Except a consumer can discard an unprocessable record? It's not the unproccessable records that are the problem it is the records that are very slow to process (for whatever reason).
drob518 11mo ago
Or it’s I/O-bound.
- jeberle 11mo ago
  That was my first reaction. It's not like Java is terribly slow, so rewriting it in a slightly faster language seems like an empty exercise.
pixelpoet 11mo ago
Yeah but how is a C++ based project supposed to hit the HN frontpage?
- selkin 11mo ago
  By having a use-after-free bug that caused a noticeable and catastrophic incident.
  - pixelpoet 11mo ago
    Like that Cloudflare outage from the other day? In any case, I'm sure there's good reason to believe that, as Rust gains popularity, there totally won't be proportionally more high profile issues in stuff made with it.
    - selkin 11mo ago
      You could write a lot of bugs in Rust. Or in any memory-safe language. Just not some bugs. It's just that starting a non-toy project using a non-safe language today is really in the "you could, but didn't stop think if you should" category.
      - pixelpoet 11mo ago
        Fair enough, and I do appreciate Rust's contributions to the spectrum of programming language considerations. Personally I'd like to become a Zig guy, though. There's also Fil-C BTW, and in normal C++ there are GCC or Clang (I forget which) extensions for detecting threading issues, even good old Valgrind is under-appreciated and under-used. In general one wants to adopt best practices and be proactive, rather than relying on the language to solve all problems (of course).

nubskr 11mo ago

In the current benchmarks, I only have Kafka and rocksdb wal, will surely try to add redpanda there as well, curious how walrus would hold up against seastar based systems.

chaotic-good 11mo ago
I don't see any mentions of p99 latency in the benchmark results. Pushing gigabytes per second is not that difficult on modern hardware. Doing so with reasonable latency is what's challenging. Also, instead of using custom benchmarks it's better to just use the OMB (open-messaging benchmark).

EdwardDiego 11mo ago

> It's the very architecture of Kafka itself due to the notorious head of line problem Except a consumer can discard an unprocessable record? I'm not certain I understand how HOL applies to Kafka, but keen to learn more :)

thinkharderdev 11mo ago
> Except a consumer can discard an unprocessable record? It's not the unproccessable records that are the problem it is the records that are very slow to process (for whatever reason).

drob518 11mo ago

Or it’s I/O-bound.

jeberle 11mo ago
That was my first reaction. It's not like Java is terribly slow, so rewriting it in a slightly faster language seems like an empty exercise.

pixelpoet 11mo ago

Yeah but how is a C++ based project supposed to hit the HN frontpage?

selkin 11mo ago
By having a use-after-free bug that caused a noticeable and catastrophic incident.
- pixelpoet 11mo ago
  Like that Cloudflare outage from the other day? In any case, I'm sure there's good reason to believe that, as Rust gains popularity, there totally won't be proportionally more high profile issues in stuff made with it.
  - selkin 11mo ago
    You could write a lot of bugs in Rust. Or in any memory-safe language. Just not some bugs. It's just that starting a non-toy project using a non-safe language today is really in the "you could, but didn't stop think if you should" category.
    - pixelpoet 11mo ago
      Fair enough, and I do appreciate Rust's contributions to the spectrum of programming language considerations. Personally I'd like to become a Zig guy, though. There's also Fil-C BTW, and in normal C++ there are GCC or Clang (I forget which) extensions for detecting threading issues, even good old Valgrind is under-appreciated and under-used. In general one wants to adopt best practices and be proactive, rather than relying on the language to solve all problems (of course).