4 ms·
Maybe you are using different terminology, because IMO you don't need "cross-service transactions to publish the event only-once" "Exactly-once" message proces
by ryanjshaw 4y ago
Maybe you are using different terminology, because IMO you don't need "cross-service transactions to publish the event only-once"
"Exactly-once" message processing is what you typically want in business operations, and that is straightforward to achieve with "at-least-once" message delivery:
1. send(message with unique .id):
a. insert into DB queue where not exists message.id
2. dispatcher:
a. fetch messages from DB queue where unpublished = 1
b. publish message to integration queue [at-least-once-delivery]
c. update message set unpublished = 0 (and ideally record some audit information e.g. the sent timestamp + Kafka message partition/offset or IBM MQ message ID)
3. receive(message):
a. check if message.id has been processed [exactly-once-processing]
b. perform operation, including linking it back to message.id, in a single DB transaction
If failure occurs:
1. between 2.a and 2.b, you fetch messages again and nothing is lost
2. between 2.b and 2.c, you republish the same message; 3.a ensures exactly-once *processing*
3. between 3.a and 3.b, you retry the message (with IBM MQ, you use transactions or peek then read; with Kafka, you don't update offsets until 3.b completes)
4. on the integration queue: set unpublished = 1 on all messages published sent in the timeframe where integration queue lost data
In practice: SQL DBs will surprise you with lock and concurrency semantics; I suggest using something battle tested under high load, or an event store database.
In a sense, this is a "cross-service transaction", but it is not a "transaction protocol", rather it is an eventually consistent design.