3 ms·
A question to answer is, what does orchestration look like if data flows are continuous, updating as quickly as possible? That's where we're headed. Suppose N s
by jgraettinger1 3y ago
A question to answer is, what does orchestration look like if data flows are continuous, updating as quickly as possible? That's where we're headed. Suppose N sources of data, M places where you want that data to be, and T streaming transformations to apply along the way.
In this world, a separate control plane handling time- and job-based polling has some architectural friction with a data-plane that wants to process records the instant they're available.
Disclosure: I have skin in the game / am co-founder of https://www.estuary.dev/ https://www.estuary.dev/