7 ms·

alex here, original author of redpanda is hard to respond to a 6-part blog series content - released all at once - on an HN thread. - what we can deterministi

alex here, original author of redpanda is hard to respond to a 6-part blog series content - released all at once - on an HN thread. - what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. - the kafka partition model of one segment per partition could be optimized in both arch - the benefit for all of us, is that all of these things will be committed to the OMB (open messaging benchmark) and will be on git for anyone interested in running it themselves. - we welcome all confluent customers (since the post is from the field cto office) to benchmark against us and choose the best platform. this is how engineering is done. In fact, we will help you run it for you at no cost. Your hardware, your workload head-to-head. We'll help you set it up with both.... but let's keep the rest of the thread technical. - log.flush.interval.messages=1 - this is something we've taken a stance a long long time ago in 2019. As someone who has personally talked to hundreds of enterprises to date, most workloads in the world should err on the side of safety and flushing to disk (fsync()). Hardware is very good today and you no longer have to choose between safety and reasonable performance. This isn't the high latency you used to see on spinning disks.

Spivak 3y ago
> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server.
- insanitybit 3y ago
  > Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.
  - Spivak 3y ago
    I suppose but that's the trade-off for performance. You have to design your system so that can't happen. Which if you're cloud then it's deploying multi-az, if you're coloing then paying for racks with separate power and/or having battery so you have time to fsync and shut down and if you're fully on-prem then you don't need my advice. Or I suppose just pay for a managed service from someone who does that for you.
  - spenczar5 3y ago
    Then enable fsync. I don't really see a way around requiring synchronization to persistent disk if you want persistence cross power outages, right?
    - insanitybit 3y ago
      Yes, that is what the linked benchmarks discuss...
  - datadeft 3y ago
    I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.
    - tyingq 3y ago
      >The original goal was to have a message queue that holds statistical data I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical". "Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."
      - datadeft 3y ago
jvanlightly 3y ago
It's a common misconception about Kafka and fsyncs. But the Kafka replication protocol has a recovery mechanism, much in the same way that Viewstamped Replication Revisited does (except it's safer due to the page cache), which allows Kafka to write to disk asynchronously. The trade-off is that we need fault domains (AZs in the cloud), but if we care about durability and availability, we should be deploying across AZs anyway. We've seen plenty of full region outages, but zero power loss events in multiple AZs in six years. Kafka and fsyncs: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...
- agallego 3y ago
  repeating things does not make them true. I read the post. You can only control some failures, but happy for us to write our thoughts in blog form.
- galeaspablo 3y ago
  Two comments here. 1) What about Kafka + KRaft, doesn't that suffer the same problem you point out in Redpanda? If so, recommending to your customers to run KRaft without fsync would be like recommending running with a Zookeeper that sometimes doesn't work. Or do I fundamentally misunderstand KRaft? 2) You mention simultaneous power failures deep into the fsync blog post. I think this should be more visible in your benchmark blog post, when you write about turnning off fsyncs.
  - ocadaruma 3y ago
    1) KRaft is only for metadata replication, and data replication is done in ISR based even in KRaft, so it doesn't change the conclusion
    - galeaspablo 3y ago
      Nah, I dug deeper into this. Right conclusion, wrong reasoning. The reason KRaft turns out to be fine is because the KRaft topic does fsync! Source: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-QuorumState https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A... KRaft is used for metadata replication in the same way that Zookeeper is used for metadata. I.e., in a very meaningful way.
dangoodmanUT 3y ago
Can you turn fsync off and rely on recovery with Redpanda?
- agallego 3y ago
  no, because it is built into the raft protocol itself. with Acks=-1 we only acknowledge to the producer once data has 1. writen to majority 2. majority has done an fsync() i can see in the future giving people opt-out options here tho.
comet-engine 3y ago
[dead]
agallego 3y ago
Following up - https://redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://redpanda.com/blog/why-fsync-is-needed-for-data-safet... Try this on your laptop see global data loss - hint: multi-az is not enough Regardless of the replication mechanism you must fsync() your data to prevent global data loss in non-Byzantine protocols.

Spivak 3y ago

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server.

insanitybit 3y ago
> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.
- Spivak 3y ago
  I suppose but that's the trade-off for performance. You have to design your system so that can't happen. Which if you're cloud then it's deploying multi-az, if you're coloing then paying for racks with separate power and/or having battery so you have time to fsync and shut down and if you're fully on-prem then you don't need my advice. Or I suppose just pay for a managed service from someone who does that for you.
- spenczar5 3y ago
  Then enable fsync. I don't really see a way around requiring synchronization to persistent disk if you want persistence cross power outages, right?
  - insanitybit 3y ago
    Yes, that is what the linked benchmarks discuss...
- datadeft 3y ago
  I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.
  - tyingq 3y ago
    >The original goal was to have a message queue that holds statistical data I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical". "Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."
    - datadeft 3y ago

jvanlightly 3y ago

It's a common misconception about Kafka and fsyncs. But the Kafka replication protocol has a recovery mechanism, much in the same way that Viewstamped Replication Revisited does (except it's safer due to the page cache), which allows Kafka to write to disk asynchronously. The trade-off is that we need fault domains (AZs in the cloud), but if we care about durability and availability, we should be deploying across AZs anyway. We've seen plenty of full region outages, but zero power loss events in multiple AZs in six years. Kafka and fsyncs: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

agallego 3y ago
repeating things does not make them true. I read the post. You can only control some failures, but happy for us to write our thoughts in blog form.
galeaspablo 3y ago
Two comments here. 1) What about Kafka + KRaft, doesn't that suffer the same problem you point out in Redpanda? If so, recommending to your customers to run KRaft without fsync would be like recommending running with a Zookeeper that sometimes doesn't work. Or do I fundamentally misunderstand KRaft? 2) You mention simultaneous power failures deep into the fsync blog post. I think this should be more visible in your benchmark blog post, when you write about turnning off fsyncs.
- ocadaruma 3y ago
  1) KRaft is only for metadata replication, and data replication is done in ISR based even in KRaft, so it doesn't change the conclusion
  - galeaspablo 3y ago
    Nah, I dug deeper into this. Right conclusion, wrong reasoning. The reason KRaft turns out to be fine is because the KRaft topic does fsync! Source: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-QuorumState https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A... KRaft is used for metadata replication in the same way that Zookeeper is used for metadata. I.e., in a very meaningful way.

dangoodmanUT 3y ago

Can you turn fsync off and rely on recovery with Redpanda?

agallego 3y ago
no, because it is built into the raft protocol itself. with Acks=-1 we only acknowledge to the producer once data has 1. writen to majority 2. majority has done an fsync() i can see in the future giving people opt-out options here tho.

comet-engine 3y ago

[dead]

agallego 3y ago

Following up - https://redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols https://redpanda.com/blog/why-fsync-is-needed-for-data-safet... Try this on your laptop see global data loss - hint: multi-az is not enough Regardless of the replication mechanism you must fsync() your data to prevent global data loss in non-Byzantine protocols.