5 ms·

I can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement. Skimming the repository

I can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement. Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.

fnord123 6y ago
According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.
- agallego 6y ago
  Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy. The reads are more complex since it involves understanding io priority scheduling etc
  - fnord123 6y ago
    >write behind strategy You ack writes before they're written?
    - agallego 6y ago
      not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
      - fnord123 6y ago
        afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't. On the other hand, the storage device can fail so what does it even mean to have written the data? :)
        agallego 6y ago
        our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
    - rystsov 6y ago
      Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.

fnord123 6y ago

According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space): https://vectorized.io/dataengineering-podcast-interview/ https://vectorized.io/dataengineering-podcast-interview/ The throughput is 4x, the latency is 10x.

agallego 6y ago
Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy. The reads are more complex since it involves understanding io priority scheduling etc
- fnord123 6y ago
  >write behind strategy You ack writes before they're written?
  - agallego 6y ago
    not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
    - fnord123 6y ago
      afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't. On the other hand, the storage device can fail so what does it even mean to have written the data? :)
      - agallego 6y ago
        our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
  - rystsov 6y ago
    Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.