4 ms·

Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" m

Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.

klabb3 3y ago
There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
- bicijay 3y ago
  Could you expand on that? How would you achieve "partial order" guarantees?
  - klabb3 3y ago
    One type of partial order would be that a producer puts all the messages that are related in the same queue, so that A always precedes B. Basically the invariant becomes: If a consumer sees an event B, it will have certainly have seen the event A before that. Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db). Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
  - CogitoCogito 3y ago
    Let's say you have events coming in that result in inserts, updates, and deletes on a table with a certain primary key. Assuming no dependencies external to this table, you only need events involving a specific key to be ordered. I.e. it doesn't really matter if row_a gets updated before or after row_b. In either case, you end up with the same thing. So if you do something like kafka partitions and you send events to certain partitions based on their primary key, then those partitions will be ordered which will be enough. That doesn't fix your example of dealing with individual errors, but in many cases that's enough.
rafaelturk 3y ago
Couldn't agree more, messages should be completely agnostic from one-another. If you have a decent event-driven architecture, you don't need kafka. and you can be happy with Redis or RabbitMQ

Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.

klabb3 3y ago

There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.

bicijay 3y ago
Could you expand on that? How would you achieve "partial order" guarantees?
- klabb3 3y ago
  One type of partial order would be that a producer puts all the messages that are related in the same queue, so that A always precedes B. Basically the invariant becomes: If a consumer sees an event B, it will have certainly have seen the event A before that. Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db). Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
- CogitoCogito 3y ago
  Let's say you have events coming in that result in inserts, updates, and deletes on a table with a certain primary key. Assuming no dependencies external to this table, you only need events involving a specific key to be ordered. I.e. it doesn't really matter if row_a gets updated before or after row_b. In either case, you end up with the same thing. So if you do something like kafka partitions and you send events to certain partitions based on their primary key, then those partitions will be ordered which will be enough. That doesn't fix your example of dealing with individual errors, but in many cases that's enough.

rafaelturk 3y ago

Couldn't agree more, messages should be completely agnostic from one-another. If you have a decent event-driven architecture, you don't need kafka. and you can be happy with Redis or RabbitMQ