3 ms·
They did, and their product is great. It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). I hav
by rebanevapustus 1y ago
They did, and their product is great.
It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs).
I have made an accessible version of a subset of Differential Dataflow (DBSP) in Python right here: https://github.com/brurucy/pydbsp https://github.com/brurucy/pydbsp
DBSP is so expressive that I have implemented a fully incremental dynamic datalog engine as a DBSP program.
Think of SQL/Datalog where the query can change in runtime, and the changes themselves (program diffs) are incrementally computed: https://github.com/brurucy/pydbsp/blob/master/notebooks/datalog.ipynb https://github.com/brurucy/pydbsp/blob/master/notebooks/data...
- gunnarmorling 1y ago> It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). Flink SQL also checks that box.
- rebanevapustus 1y agoNot true. There has to be some change in the code, and they will not share the same semantics (and perhaps won't work when retractions/deletions also appear whilst streaming). And let's not even get to the leaky abstractions for good performance (watermarks et al).
- jitl 1y agoFlink SQL is quite limited compared to Feldera/DBSP or Frank’s Materialize.com, and has some correctness limitations: it’s “eventually consistent” but until you stop the data it’s unlikely to ever be actually correct when working with streaming joins. https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/ https://www.scattered-thoughts.net/writing/internal-consiste...