4 ms·
We can't have nice things because in the real world most EDA designs are pretty naive and it's painful to look at them. If your events are pure telemetry - sam
by foobiekr 4y ago
We can't have nice things because in the real world most EDA designs are pretty naive and it's painful to look at them.
If your events are pure telemetry - samples of some data, some event that is standalone like a click, etc. - then life is easy. A missed event is no big deal, most of the time, and if it is you need some message-transactional, lossless event delivery mechanism, assuming you can pay the cost in performance or $, but those are both pretty easy concepts and people can mostly go from zero to understanding them in a day.
And here is where the trouble starts.
A lot of events in the real world are actually events that represent statefulness in the form of what are basically a log of state machine transitions, even if (because people don't really know how to build such systems) the modeling is not explicit.
In an ideal world with some publisher A and some end receiver C with perhaps some number of pipeline processors B, as long as the path from A to B to C is such that no drops or re-ordering can occur, things are should be fine. A sends to B+, the last stage of B sends to C. All good.
But in the real world, flakiness exists - bugs are _common_ and non-telemetry events are a problem:
* conveyance mechanisms are not actually reliable - they tend to have bugs, and this includes favorites like Kafka
* A, B and C can each have bugs - they might drop a message, they might _do the wrong thing_, they might persist some incorrect state, etc.
This is how you get into the situation I dealt with last month where the SRE team had to occasionally fix rows in the database where due to a bug that lasted for months some rows would be wrong, and sticky wrong because the system didn't have any kind of inconsistency-discovery mechanism even though the state data was always available.
Almost every event-driven system I've seen in the real world lacks the ability to detect this even when the source of truth is available (A, for example). Even a simple re-transmission or some digest of the state to detect inconsistency between the bottom and the top layers is routinely absent. It's pretty bad.
By analogy, anyone who has done low-level work will be familiar with the very different classes of events that are usually called "edge triggered (notified once, lost after)" or "level triggered (visible for as long as the state persists)"; more broadly there are stateful and non-stateful. Level triggered is _always_ easier to deal with because you can implement periodic self-check to recover in the event that an ephemeral bug has caused the event to be lost or mis-processed. There is incredible irony in the fact that big complex EDA systems have lost the thread on what is, in fact, the easy case.