3 ms·

Rama PStates are globally readable but not globally writable. They are only writable from the topology that declares them. All code writing to PStates is thereb

Rama PStates are globally readable but not globally writable. They are only writable from the topology that declares them. All code writing to PStates is thereby always in the exact same program. Additionally, since PStates are not the source of truth – the depots (event logs) are – mistakes can be corrected via recompute from the source of truth. "Alice's current location" would be done in Rama like this: * Have a depot that receives new locations for people. The appended records would have three keys: userId, location, and timestamp. The depot is partitioned by userId. * Have an ETL with a PState called $$currentLocation that's a map from userId to location. * The ETL consumes the depot and update the PState as new data comes in.

fipar 3y ago
Thanks for the answer. So with a depot being responsible for locations, and partitioned by user id, is is too off to think of this as the way cockroachdb does sharding, with ranges where there's a single raft leader per range? From your description of PStates, I reckon there's some inherent delay from when "an event happens" (bear with me, I know ...), the depot appends it to its log/stream, and a PState consumes it? Let me try to make a concrete example: suppose I'm a consumer that will make a decision based on Alice's location (it could be a media streaming service that must offer a different catalog view depending on the region the user is, even for the same user). What's the Rama way to know how good my location knowledge of Alice is (e.g., "I can be sure I know where Alice was no longer than T time ago")?
- nathanmarz 3y ago
  When you use stream ETLs, the delay between a depot append and the corresponding PState updates becoming visible is in the single-digit millis range. With microbatching it's at least a few hundred millis. Coordination of understanding whether your writes have propagated to PStates in consuming ETLs is provided at the depot append level. If you do depot appends with full acking enabled (which is the default), the append call doesn't complete until all colocated stream topologies have finished processing that record. So in your client code (using Clojure here for the example), you could do: (foreign-append! locations-depot {:user-id alice-id :location "NYC" :timestamp 12345}) (foreign-select-one (keypath alice-id) current-locations-pstate) That PState query is guaranteed to include the depot write that was done prior.

fipar 3y ago

Thanks for the answer. So with a depot being responsible for locations, and partitioned by user id, is is too off to think of this as the way cockroachdb does sharding, with ranges where there's a single raft leader per range? From your description of PStates, I reckon there's some inherent delay from when "an event happens" (bear with me, I know ...), the depot appends it to its log/stream, and a PState consumes it? Let me try to make a concrete example: suppose I'm a consumer that will make a decision based on Alice's location (it could be a media streaming service that must offer a different catalog view depending on the region the user is, even for the same user). What's the Rama way to know how good my location knowledge of Alice is (e.g., "I can be sure I know where Alice was no longer than T time ago")?

nathanmarz 3y ago
When you use stream ETLs, the delay between a depot append and the corresponding PState updates becoming visible is in the single-digit millis range. With microbatching it's at least a few hundred millis. Coordination of understanding whether your writes have propagated to PStates in consuming ETLs is provided at the depot append level. If you do depot appends with full acking enabled (which is the default), the append call doesn't complete until all colocated stream topologies have finished processing that record. So in your client code (using Clojure here for the example), you could do: (foreign-append! locations-depot {:user-id alice-id :location "NYC" :timestamp 12345}) (foreign-select-one (keypath alice-id) current-locations-pstate) That PState query is guaranteed to include the depot write that was done prior.