3 ms·
In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the componen
by squeaky-clean 3y ago
In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the components it's describing seem to be equivalent to what he calls the Batch Layer and the Serving Layer in his book.
But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.
- nathanmarz 3y agoRama codifies and integrates the concepts I described in my book, with the high level model being: indexes = function(data) and query = function(indexes). These correspond to "depots" (data) , "ETLs" (functions), "PStates" (indexes), and "queries" (functions). Rama is not batch-based. That is, PStates are not materialized by recomputing from scratch. They're incrementally updated either with stream or microbatch processing. But PStates can be recomputed from the source data on depots if needed.
- Pxtl 3y agoSo the idea is that you could do 1. send event data to depot 2. trigger localized ETLs (or put it high-priority in queue) to recalculate just the impacted data into relevant PStates 3. await completion of aforementioned ETLs 4. run query from updated PStates Maybe too heavy for an upvote, but very appropriate for a an important transaction like a purchase.
- FridgeSeal 3y agoForgive me if I’m misunderstanding things, but this seems quite similar to what Materialize and ReadySet do, but like “as a library”, because Rama doesn’t use a “separate” layer for the storage stuff. Is that correct-ish?