3 ms·

Oh, it took us around 2 years to have somewhat reliable event dispatching with transactional outboxes. We've run into so many edge cases under highload: - usin

by kgeist 4y ago

Oh, it took us around 2 years to have somewhat reliable event dispatching with transactional outboxes. We've run into so many edge cases under highload: - using the autoincrement ID as a pointer to the "last processed event" not knowing that MySQL reserves IDs non-atomically with inserts so it's possible for the relay to "see" an event with a bigger ID before an event with a smaller ID becomes visible, thus skipping some events - implementing event handlers with "only once" semantics instead of "at least once" (retries on error corrupting data) - event queue processing completely halting for the entire application when an event handler throws an error due to a bug and so gets retried infinitely - some other race conditions in the implementation when the order of events gets messed up - too fine-grained events without batching overflowing the queue (takes too much time for an event to get processed) - the relay getting OOMs due to excessive batching - once we had a funny bug when code which updated the last processed ID of the current DB shard (each client has their own shard) wrote to the wrong shards and so our relay started replaying thousands events from years ago - some event handlers always sending mail as part of processing events, so when it's retried on error or replayed (see the bug above) clients receive same emails multiple times And still we have sometimes weird bugs like once a month for some reason we see a random event getting replayed in a deleted account, still tracking it down.

ryanjshaw 4y ago
Nice list, brings back fond memories. On MS SQL you have a similar scenario IIRC - you can READ COMMITTED to block the query when there are dirty writes in the range of rows, but you're also encouraged in general to actually use READ COMMITTED SNAPSHOT, which won't block... And then you get the problem where a different thread is inserting a message into the queue table, but it's taking a while, and the dispatcher is reading the queue table, and misses the uncommitted message... and never sees it again because you're using "last processed offset" as your "resume point", and you've moved past that point by the time the concurrent message insert completes. That's why I use "IsUnpublished = 1" as my flag, which also makes replay very simple to do (just create a filtered index on that column for performance reasons). When I first started building applications like this, I eventually learned that the people discussing these patterns placed very little explicit emphasis on distinguishing between "application events" and "integration events" -- if you are struggling with too many events, maybe this is where your issue is. A final point - your business logic event handler shouldn't send mail. You should produce an "operation completed (ID)" event and send the email off the back of that in an email notification event handler.

by kgeist 4y ago

ryanjshaw 4y ago

Nice list, brings back fond memories. On MS SQL you have a similar scenario IIRC - you can READ COMMITTED to block the query when there are dirty writes in the range of rows, but you're also encouraged in general to actually use READ COMMITTED SNAPSHOT, which won't block... And then you get the problem where a different thread is inserting a message into the queue table, but it's taking a while, and the dispatcher is reading the queue table, and misses the uncommitted message... and never sees it again because you're using "last processed offset" as your "resume point", and you've moved past that point by the time the concurrent message insert completes. That's why I use "IsUnpublished = 1" as my flag, which also makes replay very simple to do (just create a filtered index on that column for performance reasons). When I first started building applications like this, I eventually learned that the people discussing these patterns placed very little explicit emphasis on distinguishing between "application events" and "integration events" -- if you are struggling with too many events, maybe this is where your issue is. A final point - your business logic event handler shouldn't send mail. You should produce an "operation completed (ID)" event and send the email off the back of that in an email notification event handler.