3 ms·

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluen

by Spivak 4y ago

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server.

insanitybit 4y ago
> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.
- Spivak 4y ago
  I suppose but that's the trade-off for performance. You have to design your system so that can't happen. Which if you're cloud then it's deploying multi-az, if you're coloing then paying for racks with separate power and/or having battery so you have time to fsync and shut down and if you're fully on-prem then you don't need my advice. Or I suppose just pay for a managed service from someone who does that for you.
- spenczar5 4y ago
  Then enable fsync. I don't really see a way around requiring synchronization to persistent disk if you want persistence cross power outages, right?
  - insanitybit 4y ago
    Yes, that is what the linked benchmarks discuss...
- datadeft 4y ago
  I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.
  - tyingq 4y ago
    >The original goal was to have a message queue that holds statistical data I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical". "Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."
    - datadeft 4y ago
      I guess you can add any feature to anything. I think this whole investor driven development is just sad.
  - ako 4y ago
    Do you have a source/link for that original goal? I wasn’t aware of this, and as such expect that I can rely on kafka for my events. Also, if this is really the case it should be mentioned on the homepage of Kafka. Just checked kafka’s homepage, it mentions mission critical, durable, fault tolerant, stores data safely, zero message loss, trusted… Seems they’ve moved on from their original goal.
    - datadeft 4y ago
      "The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type." https://kafka.apache.org/ https://kafka.apache.org/
  - skrtskrt 4y ago
    Kafka is used widely as a persistent event store, and its development features reflect that. Why would I not just turn on fsync or deploy in a distributed pattern for reliability so I can just continue using it instead of ripping it out, benchmarking something new, teaching the entire org something new, potentially negotiating a new contract, and then executing a huge migration?
- threeseed 4y ago
  It's not a serious problem for most deployments though. You should be running Kafka in multiple DCs/AZs for high availability and scalability. And in that scenario fsync is nice but not necessary.

by Spivak 4y ago

insanitybit 4y ago

> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.

Spivak 4y ago
I suppose but that's the trade-off for performance. You have to design your system so that can't happen. Which if you're cloud then it's deploying multi-az, if you're coloing then paying for racks with separate power and/or having battery so you have time to fsync and shut down and if you're fully on-prem then you don't need my advice. Or I suppose just pay for a managed service from someone who does that for you.
spenczar5 4y ago
Then enable fsync. I don't really see a way around requiring synchronization to persistent disk if you want persistence cross power outages, right?
- insanitybit 4y ago
  Yes, that is what the linked benchmarks discuss...
datadeft 4y ago
I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.
- tyingq 4y ago
  >The original goal was to have a message queue that holds statistical data I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical". "Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."
  - datadeft 4y ago
    I guess you can add any feature to anything. I think this whole investor driven development is just sad.
- ako 4y ago
  Do you have a source/link for that original goal? I wasn’t aware of this, and as such expect that I can rely on kafka for my events. Also, if this is really the case it should be mentioned on the homepage of Kafka. Just checked kafka’s homepage, it mentions mission critical, durable, fault tolerant, stores data safely, zero message loss, trusted… Seems they’ve moved on from their original goal.
  - datadeft 4y ago
    "The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type." https://kafka.apache.org/ https://kafka.apache.org/
- skrtskrt 4y ago
  Kafka is used widely as a persistent event store, and its development features reflect that. Why would I not just turn on fsync or deploy in a distributed pattern for reliability so I can just continue using it instead of ripping it out, benchmarking something new, teaching the entire org something new, potentially negotiating a new contract, and then executing a huge migration?
threeseed 4y ago
It's not a serious problem for most deployments though. You should be running Kafka in multiple DCs/AZs for high availability and scalability. And in that scenario fsync is nice but not necessary.