We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- wit

We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- with an important twist. The central concept is a "collection", an append-only set of schematized documents, which can be captured and materialized into other systems (e.x. pub/sub, S3 buckets, etc). "Derivations" are collections defined in terms of source collections, and stateful transformations/joins/aggregations which are applied to them. A key twist is that collections are simultaneously a batch dataset (backed by cloud-storage) and also a real-time stream. They unify the current dichotomy of "historical" vs "streaming" data into a single addressed concept. Declare a new derivation, and it automatically back-fills over history right from S3, then seamlessly transitions to live data. If this sounds interesting, check out our docs [0]. We're early, but love feedback! [0] https://estuary.readthedocs.io/en/latest/README.html https://estuary.readthedocs.io/en/latest/README.html