3 ms·

A new McSharry post! Excellent Last I checked, VMWare had moved away from differential datalog?

jitl 1y ago
The Differential Datalog team founded Feldera: https://www.feldera.com/ https://www.feldera.com/ They switched from differential Datalog to differential SQL, I think because they realized Datalog is a really tough sell.
- rebanevapustus 1y ago
  They did, and their product is great. It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). I have made an accessible version of a subset of Differential Dataflow (DBSP) in Python right here: https://github.com/brurucy/pydbsp https://github.com/brurucy/pydbsp DBSP is so expressive that I have implemented a fully incremental dynamic datalog engine as a DBSP program. Think of SQL/Datalog where the query can change in runtime, and the changes themselves (program diffs) are incrementally computed: https://github.com/brurucy/pydbsp/blob/master/notebooks/datalog.ipynb https://github.com/brurucy/pydbsp/blob/master/notebooks/data...
  - gunnarmorling 1y ago
    > It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). Flink SQL also checks that box.
    - rebanevapustus 1y ago
      Not true. There has to be some change in the code, and they will not share the same semantics (and perhaps won't work when retractions/deletions also appear whilst streaming). And let's not even get to the leaky abstractions for good performance (watermarks et al).
    - jitl 1y ago
      Flink SQL is quite limited compared to Feldera/DBSP or Frank’s Materialize.com, and has some correctness limitations: it’s “eventually consistent” but until you stop the data it’s unlikely to ever be actually correct when working with streaming joins. https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/ https://www.scattered-thoughts.net/writing/internal-consiste...

jitl 1y ago

The Differential Datalog team founded Feldera: https://www.feldera.com/ https://www.feldera.com/ They switched from differential Datalog to differential SQL, I think because they realized Datalog is a really tough sell.

rebanevapustus 1y ago
They did, and their product is great. It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). I have made an accessible version of a subset of Differential Dataflow (DBSP) in Python right here: https://github.com/brurucy/pydbsp https://github.com/brurucy/pydbsp DBSP is so expressive that I have implemented a fully incremental dynamic datalog engine as a DBSP program. Think of SQL/Datalog where the query can change in runtime, and the changes themselves (program diffs) are incrementally computed: https://github.com/brurucy/pydbsp/blob/master/notebooks/datalog.ipynb https://github.com/brurucy/pydbsp/blob/master/notebooks/data...
- gunnarmorling 1y ago
  > It is the only database/query engine that allows you to use the same SQL for both batch and streaming (with UDFs). Flink SQL also checks that box.
  - rebanevapustus 1y ago
    Not true. There has to be some change in the code, and they will not share the same semantics (and perhaps won't work when retractions/deletions also appear whilst streaming). And let's not even get to the leaky abstractions for good performance (watermarks et al).
  - jitl 1y ago
    Flink SQL is quite limited compared to Feldera/DBSP or Frank’s Materialize.com, and has some correctness limitations: it’s “eventually consistent” but until you stop the data it’s unlikely to ever be actually correct when working with streaming joins. https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/ https://www.scattered-thoughts.net/writing/internal-consiste...