4 ms·
Remember that both Kafka and Kinesis have "at least once" delivery and from experience it's a common scenario. You will receive multiple messages. Two options:
by threeseed 9y ago
Remember that both Kafka and Kinesis have "at least once" delivery and from experience it's a common scenario. You will receive multiple messages. Two options: (1) store hashes of each row and then deduplicate, (2) use a database.
I can't remember if it's an issue for change capture since in theory you have some insert timestamp column anyway.
- cmatta 9y agoKafka has "exactly once" delivery guarantees since 0.11, I believe this will be the default in Kafka 1.0 (I work for Confluent).