6 ms·
Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, t
by trimbo 6y ago
Consider processing two events without guaranteed ordering:
- Create A
- Delete A
In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.
- tyrust 6y agoDoes adding timestamps not handle this case?
- jedmeyers 6y agoIf events are generated by different processes you cannot really guarantee that time is exactly the same for them, unless you do something fancy to ensure that.
- zaphirplane 6y agoInteresting. The ordering here is when the event was generated or when the event entered the queue ? I think the later and so I think the examples here don’t apply without something on top and a trade off
- jedmeyers 6y agoThe queue entrypoint is not always the same process either, especially in a system like Pub/Sub.
- jurre 6y agoNow you need to queue up events for some time, reorder them using the timestamp, and then process them. It’s possible, but has overhead in both performance and custom code you’ll have to maintain. If there is no guarantee of order, two separate systems consuming those same events also might get different results, depending on the implementation, that can be problematic
- valleyjo 6y agoThat won’t work in all cases. For instance, if you get messages from devices which can be reimaged they may have clock skew in a period of time before they’re synchronized again.
- easytiger 6y agoBut in any case you can't rely on the order of message ingress to your system to represent anything meaningful either? It would have to ensure that the key for defining order would have some hard logical ordering purpose for which time is not relevant or useful.
- jlokier 6y agoThe order of message ingress can still be meaningful even if device clocks are skew or jump due to rebooting, reimaging, network time sync, frequency drift, etc. A hard logical order arises from interactions. E.g. if the device receives a message, does something locally, goes through a clock change, and then sends a message dependent on one it fetched earlier, that's a logical order with out-of-order clock. Or if a device gets a message, processes, sends something to another device, that one processes too then sends another message back to the original source, there's a logical order but with three different clocks. Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order.
- easytiger 6y agoThat guarantee/assumption can never really be made. Message A on event a' from system a might be sent to system b effecting event b' and thus message B to be sent by <insert medium here>, and consumed and correlated by consumer software MC on hardware mc. However system B might take longer to flush it's hardware/software buffer and the message arrives at mc before message A, for example. I've encountered this many times. That data has no meaning in itself except in the meta. > Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order. If you are consuming from sources which you cannot control the accuracy of the clocks, then you must inherently either reduce your reliance on the need for accuracy (many Windows systems have horrendous clock discipline in their software implementation) or find a way of ensuring accuracy. E.g. close proximity NTP or close proximity PTP etc etc. Hope that makes sense.
- GordonS 6y agoWith a single producer and consumer, yes - but of course that's seldom the case. With multiple producers and consumers, clock skew would be an issue, with the time on different machines being off from each other slightly. One option is to use a single source for generating IDs, but that introduces another failure point, and comes at a hefty performance cost.
- dragonwriter 6y ago> Does adding timestamps not handle this case? If you have one message source (a single thread or some kind of coordination), and the messages have lower frequency than the timestamp resolution, yes. The farther you get from that, the more the answer is no.
- sumtechguy 6y agoFor a single process on one box with one thread you can use something like that. If you involve more than 1 box that goes out the window. Sometimes you can still get 'one timestamp' by making something else the owner of the timestamp. It also depends on your resolution of time and the process that does the ingesting. For example if that ingest process has more than one thread to handle things you can still get out of order/sametimestamp if not coded correctly.
- chimera77 6y agoThere are a couple of options without needing guaranteed ordering: - jobs can have ever increasing ids, workers record the last seen id in one place, and ignore jobs with ids less than last seen - job results are returned for each job to a supervisor. if a job result doesn't match current expected state, resend job. jobs should be idempotent in case a job is sent multiple times If the create job is expensive, the latter solution would be less ideal, though.
- manigandham 6y agoJust ignoring tasks isn't helpful, it means you still process things out of order and just never get to the previous pending item in that case. Much easier to just have strict order and process it off the line as it comes.
- bgdam 6y agoCreating ever increasing ids reliably at scale is not trivial. You will probably end up having a single server generating these ids which will then become a single point of failure.
- caturopath 6y ago(Or a distributed system - this is no trickier to migrate away from SPOF than anything else with global state.)
- zaphirplane 6y agoBut when does the delete enter the queue and when does the Create enter the queue ;)
- inopinatus 6y agoThat doesn’t necessarily require that the entire queue is totally ordered, but the alternatives (such as Virtual Synchrony) are still considered arcane / research topics.
- staticassertion 6y agoKinda interesting, I built this just yesterday but with an async "buffer" and "write", and I just used a simple incrementing identifier system.