4 ms·
When you use stream ETLs, the delay between a depot append and the corresponding PState updates becoming visible is in the single-digit millis range. With micro
by nathanmarz 3y ago
When you use stream ETLs, the delay between a depot append and the corresponding PState updates becoming visible is in the single-digit millis range. With microbatching it's at least a few hundred millis.
Coordination of understanding whether your writes have propagated to PStates in consuming ETLs is provided at the depot append level. If you do depot appends with full acking enabled (which is the default), the append call doesn't complete until all colocated stream topologies have finished processing that record. So in your client code (using Clojure here for the example), you could do:
(foreign-append! locations-depot {:user-id alice-id :location "NYC" :timestamp 12345})
(foreign-select-one (keypath alice-id) current-locations-pstate)
That PState query is guaranteed to include the depot write that was done prior.