4 ms·

I would love if something like this that exposed C bindings so that every language with an FFI could use the library. I’d love to be able to define pipelines a

I would love if something like this that exposed C bindings so that every language with an FFI could use the library. I’d love to be able to define pipelines and queries in .NET instead of having to use SQL.

lsuresh 2y ago
Hi Nelkins. We do have a Rust crate you could consider using directly: https://docs.rs/dbsp/latest/dbsp/ https://docs.rs/dbsp/latest/dbsp/. Our SQL compiler puts together a pipeline by generating a Rust program that uses this crate.
- nextaccountic 2y ago
  How do you compare the dbsp crate with other libraries for incremental computation, like salsa [0] and adapton [1]? Could dbsp work as a backend for an incremental compiler? [0] https://github.com/salsa-rs/salsa https://github.com/salsa-rs/salsa [1] https://docs.rs/adapton/latest/adapton/ https://docs.rs/adapton/latest/adapton/
  - lsuresh 2y ago
    They're quite different from DBSP. Given a program execution, Salsa/adapton seem to reuse prior steps from an execution when the input changes. In contrast, DBSP has built-in knowledge of incremental versions of operations and composes them. Here's a blurb from a recent paper we submitted: DBSP is not tied to databases in any way; it is in fact a Turing-complete language that can be used for many other purposes. But it works particularly well in the area of databases, for two reasons: – DBSP operates on values from a commutative group. Databases can be modeled as a commutative group. – DBSP reduces the problem of incrementalizing a complex program to the problem of incrementalizing each primitive operation that appears in the program. For databases there are known efficient incremental implementations for all primitive operations. As to whether DBSP could be used for an incremental compiler backend, we think not (at least not with our current understanding). Would be cool to explore though.
loxias 2y ago
Second the desire for C bindings! (or someone showing how to wrap and call the rust bindings?)
- lsuresh 2y ago
  The previous implementation we built at VMware went from datalog -> Rust, and we supported other language bindings using C bindings and FFI. The same ought to work here too.

lsuresh 2y ago

Hi Nelkins. We do have a Rust crate you could consider using directly: https://docs.rs/dbsp/latest/dbsp/ https://docs.rs/dbsp/latest/dbsp/. Our SQL compiler puts together a pipeline by generating a Rust program that uses this crate.

nextaccountic 2y ago
How do you compare the dbsp crate with other libraries for incremental computation, like salsa [0] and adapton [1]? Could dbsp work as a backend for an incremental compiler? [0] https://github.com/salsa-rs/salsa https://github.com/salsa-rs/salsa [1] https://docs.rs/adapton/latest/adapton/ https://docs.rs/adapton/latest/adapton/
- lsuresh 2y ago
  They're quite different from DBSP. Given a program execution, Salsa/adapton seem to reuse prior steps from an execution when the input changes. In contrast, DBSP has built-in knowledge of incremental versions of operations and composes them. Here's a blurb from a recent paper we submitted: DBSP is not tied to databases in any way; it is in fact a Turing-complete language that can be used for many other purposes. But it works particularly well in the area of databases, for two reasons: – DBSP operates on values from a commutative group. Databases can be modeled as a commutative group. – DBSP reduces the problem of incrementalizing a complex program to the problem of incrementalizing each primitive operation that appears in the program. For databases there are known efficient incremental implementations for all primitive operations. As to whether DBSP could be used for an incremental compiler backend, we think not (at least not with our current understanding). Would be cool to explore though.

loxias 2y ago

Second the desire for C bindings! (or someone showing how to wrap and call the rust bindings?)

lsuresh 2y ago
The previous implementation we built at VMware went from datalog -> Rust, and we supported other language bindings using C bindings and FFI. The same ought to work here too.