3 ms·

Thanks for the answer. So with a depot being responsible for locations, and partitioned by user id, is is too off to think of this as the way cockroachdb does

by fipar 3y ago

Thanks for the answer. So with a depot being responsible for locations, and partitioned by user id, is is too off to think of this as the way cockroachdb does sharding, with ranges where there's a single raft leader per range? From your description of PStates, I reckon there's some inherent delay from when "an event happens" (bear with me, I know ...), the depot appends it to its log/stream, and a PState consumes it? Let me try to make a concrete example: suppose I'm a consumer that will make a decision based on Alice's location (it could be a media streaming service that must offer a different catalog view depending on the region the user is, even for the same user). What's the Rama way to know how good my location knowledge of Alice is (e.g., "I can be sure I know where Alice was no longer than T time ago")?

nathanmarz 3y ago
When you use stream ETLs, the delay between a depot append and the corresponding PState updates becoming visible is in the single-digit millis range. With microbatching it's at least a few hundred millis. Coordination of understanding whether your writes have propagated to PStates in consuming ETLs is provided at the depot append level. If you do depot appends with full acking enabled (which is the default), the append call doesn't complete until all colocated stream topologies have finished processing that record. So in your client code (using Clojure here for the example), you could do: (foreign-append! locations-depot {:user-id alice-id :location "NYC" :timestamp 12345}) (foreign-select-one (keypath alice-id) current-locations-pstate) That PState query is guaranteed to include the depot write that was done prior.