3 ms·
Apache Flink does provide end-to-end exactly-once guarantees when coupled with data sources and data sinks that participate in its checkpointing mechanism. See:
by yamrzou 2y ago
Apache Flink does provide end-to-end exactly-once guarantees when coupled with data sources and data sinks that participate in its checkpointing mechanism. See:
- An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!) — https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/ https://flink.apache.org/2018/02/28/an-overview-of-end-to-en...
- Flink's Fault Tolerance Guarantees — https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/guarantees/ https://nightlies.apache.org/flink/flink-docs-release-1.20/d...
- shepherdjerred 2y agoExactly once processing != exactly once delivery
- ahoka 2y agoBut it’s mostly the processing that is interesting. Like this message I write now might see several retransmissions until you read it, but it’s a single message.
- frant-hartm 2y agoAnd on top of that this _end-to-end exactly-once guarantees_ makes the result look like the message was processed in the pipeline only once, while in reality a message will be processed 2x (or more times) in the pipeline in case of a temporary failure but only one version will be committed to the external system.