4 ms·
> At my last company I built an ordered transaction outbox to handle this. Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, Active
by paulddraper 3y ago
> At my last company I built an ordered transaction outbox to handle this.
Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis).
If you use one of these, it's easy to solve ordering.
- skeeter2020 3y agoThis is just for arrival order though. The GP mentions transaction-based operations; handling this without massive hits to parallel processing is not an easy problem to solve.
- paulddraper 3y ago? I'm not sure I understand. You choose an FIFO key, e.g. order ID. Any message with that key is processed serially. Ordering between keys is unspecified. It is important to choose a key that guarantees sufficient ordering (for correctness), but not too much (for performance).
- ignoramous 3y ago> You choose an FIFO key, e.g. order ID. This is the part of the system that is doing most of the heavy lifting. Ex A, The struggles of Elasticsearch: https://archive.is/rm1UI https://archive.is/rm1UI
- sarchertech 3y agoIf you only use one of those, for everything it’s easier. I still wouldn’t say it’s easy because it requires thought, and in my experience is commonly done incorrectly. I’ve regularly seen anything from picking the wrong partition key, to grabbing a bunch of messages from a single partition to process concurrently. If you have an existing system where some services are fully event sourced, or some depend on complicated database transactions but still want to publish to a Kafka topic etc…, it gets more complicated.