4 ms·

Nice! It'd be great to know where do the improvements came from. Is it from a different architectural design, is it from better exploitation of new hardware,

Nice! It'd be great to know where do the improvements came from. Is it from a different architectural design, is it from better exploitation of new hardware, is it from JVM limitations? For example, when Valhalla and Loom land in the JDK, how much of this improvement will stay?

fnord123 6y ago
I can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement. Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.
- fnord123 6y ago
  According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.
  - agallego 6y ago
    Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy. The reads are more complex since it involves understanding io priority scheduling etc
    - fnord123 6y ago
      >write behind strategy You ack writes before they're written?
      - agallego 6y ago
        not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
        fnord123 6y ago
        afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't. On the other hand, the storage device can fail so what does it even mean to have written the data? :)
        agallego 6y ago
        our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
      - rystsov 6y ago
        Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.
steeve 6y ago
Better exploiting hardware (via seastar and the one-thread-per-core model, see ScyllaDB vs Cassandra also) and probably better architecture.
- BenoitP 6y ago
  This is sort of what I'm getting at. The one-thread-per-core part should bring little to no benefit vs JDK+Loom, for example.
  - agallego 6y ago
    Its a bit deeper. Custom strategies for reading and writing have a huge impact. (orders of magnitude for specific workloads rather than generic page cache scheduling) Optimizing for fragmented buffers. Etc. See the LinkedIn vid from the blog posts for more low level deets.
agallego 6y ago
Sure. Gists are No use of page cache instead custom built read ahead and write behind strategy Io scheduling groups from seastar Custom intra cluster RPC to optimize for fragmented buffers native support (most storage work is small materialized metadata and large blob) Thread per core design with zero locks on the hotpath If you watch our chat with the LinkedIn kafka team, you will learn a bit more too Hopefully this helps

fnord123 6y ago

I can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement. Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.

fnord123 6y ago
According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.
- agallego 6y ago
  Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy. The reads are more complex since it involves understanding io priority scheduling etc
  - fnord123 6y ago
    >write behind strategy You ack writes before they're written?
    - agallego 6y ago
      not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
      - fnord123 6y ago
        afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't. On the other hand, the storage device can fail so what does it even mean to have written the data? :)
        agallego 6y ago
        our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
    - rystsov 6y ago
      Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.

steeve 6y ago

Better exploiting hardware (via seastar and the one-thread-per-core model, see ScyllaDB vs Cassandra also) and probably better architecture.

BenoitP 6y ago
This is sort of what I'm getting at. The one-thread-per-core part should bring little to no benefit vs JDK+Loom, for example.
- agallego 6y ago
  Its a bit deeper. Custom strategies for reading and writing have a huge impact. (orders of magnitude for specific workloads rather than generic page cache scheduling) Optimizing for fragmented buffers. Etc. See the LinkedIn vid from the blog posts for more low level deets.

agallego 6y ago

Sure. Gists are No use of page cache instead custom built read ahead and write behind strategy Io scheduling groups from seastar Custom intra cluster RPC to optimize for fragmented buffers native support (most storage work is small materialized metadata and large blob) Thread per core design with zero locks on the hotpath If you watch our chat with the LinkedIn kafka team, you will learn a bit more too Hopefully this helps