Having done several event-processing systems (lately with Storm in Java/Clojure and Onyx in Clojure), I fail any event outside the hotpath, which sends it to an

Having done several event-processing systems (lately with Storm in Java/Clojure and Onyx in Clojure), I fail any event outside the hotpath, which sends it to an alternate queue where remediation passes attempt processing. This results in slightly delayed processing (we're talking a handful of seconds max), but keeps the high-throughput hot path very simple. Events that can't be handled in the secondary queue are logged/journaled and a report is generated periodically when such events exist. It's fairly rare that I would even attempt to track internal state in an event-processing system (a node will typically emit/attach all necessary information for the next node), but in cases where I do (real-time numerical calculations), we accept a small percentage of error. We'd typically write a health check around our confidence in that calculation and expose that to systems that need to interact with it. In terms of errors that are systemic rather than due to malformed events, I'd reprocess out of the queue assuming the timing makes sense. For nodes that care about time, we'd write in checks using the event's timestamp as a guard against processing (failing events outside the time range).