5 ms·

> but it is more general, because it also lets you express relations between tables themselves (e.g. this "table" is built from the relationship between two sma

> but it is more general, because it also lets you express relations between tables themselves (e.g. this "table" is built from the relationship between two smaller tables), and it does so without requiring extra special case semantics like VIEWs. If I understand you correctly, you can easily get the same with ephemeral models in dbt or CTEs generally? > Because of this, it's easy to write small fragments of Datalog programs, and then stick it together with other fragments, without a lot of planning ahead of time, meaning as a language it is very compositional. This can be a benefit in some cases, I guess, but how can you guarantee correctness with flexibility involved? With SQL, I get either table or column level lineage with all modern tools, can audit each upstream output before going into a downstream input. In dbt I have macros which I can reuse everywhere. It's very compositional while at the same time perfectly documented and testable at runtime. Could you share a more specific example or scenario where you have seen Datalog/ Logica outperform a modern SQL setup? Generally curious. I am not at all familiar with the Logica/Datalog/Prolog world.

from-nibly 2y ago
Prolog et al is a real brain buster. As in it will break your spirits and build you back up better. I remember in college I was able to build a binary tree with 3 lines of code. And once you write the insert, the delete, search, and others just magically appear. It also frames your thinking about defining what you want rather than how to get it. If you really want to see the power of these kinds of languages look up Einstein's puzzle solved with prolog. The solution just magically comes out by entering the constraints of the puzzle.
- rytis 2y ago
  I suppose something like this: https://stackoverflow.com/a/8270393 https://stackoverflow.com/a/8270393 ?
- surgical_fire 2y ago
  I had to use Prolog in college, and while I never saw it in the wild - I at least never stumbled upon a scenario where prolog was the answer - I really enjoyed how I had to change how I looked at a problem in order to solve it in prolog.
burakemir 2y ago
Here is a proof that you can translate non-recursive datalog into relational algebra and vice versa: https://github.com/google/mangle/blob/main/docs/spec_explain_relational_algebra.md https://github.com/google/mangle/blob/main/docs/spec_explain... Since Logica is translated to SQL it should benefit from all the query optimistic goodness that went into the SQL engine that runs the resulting queries. I personally see the disadvantages of SQL in that it is not really modular, you cannot have libraries, tests and such. Disclosure: I wrote Mangle (the link goes to the Mangle repo), another datalog, different way of extending, no SQL translation but an engine library.
- aseipp 2y ago
  Mangle looks very interesting, thanks for the share. In particular I love your GRPC demo, because it shows a prototype of something I've been thinking about for a long time: what if we did GraphQL, but with Datalog! Maybe we could call it LogiQL :) In particular many people talk a lot about concerns like optimizations across GraphQL plans and how they are expected to behave on underlying tables, but this is something that I think has seen a lot of research in the Datalog realm. And to top it off, even ignoring that, Datalog just feels much more natural to write and read after a bit of practice, I think. (Obviously you need to be in the pure fragment of datalog without recursion, but even then it might be feasible to add those features with termination criteria even if it's just "decrement an internal counter and if it hits zero throw a big error") What do you think the plans for the Rust implementation will be? That's probably the most likely place I'd use it, as I don't really use Go that much.
  - burakemir 2y ago
    The Mangle repo has the beginnings of a Rust implementation but it will take some time before it is usable. The go implementation is also still being improved, but I think real DB work with persistent data will happen only in Rust. Bindings to other host languages would also use the Rust implementation. There are no big challenges here it is just work and takes time. The combination of top-down and bottom up logic programming is interesting, especially when one can move work between pre computation and query time. I like that optimizing queries in datalog can be discussed like optimization of programming language but of course the biggest gains in DB come from join order and making use of indices. There is a tension here between declarative and having some control or hints for execution. I haven't yet figured out how one should go about it, and also how to help programmers combine top-down and bottom-up computation. Work in progress! :-)
aseipp 2y ago
> If I understand you correctly, you can easily get the same with ephemeral models in dbt or CTEs generally? You can bolt on any number of 3rd party features or extensions to get some extra thing, that goes for any tool in the world. The point of something like Datalog is that it can express a similar class of relational programs that SQL can, but with a smaller set of core ideas. "Do more with less." > I guess, but how can you guarantee correctness with flexibility involved? How do you guarantee the correctness of anything? How do you know any SQL query you write is correct? Well, as the author, you typically have a good idea. The point of being compositional is that it's easier to stick together arbitrary things defined in Datalog, and have the resulting thing work smoothly. Going back to the previous example, you can define any two "tables" and then just derive a third "table" from these, using language features that you already use -- to define relationships between rows. Datalog can define relations between rules (tables) and between facts (rows), all with a single syntactic/semantic concept. While SQL can only by default express relations between rows. Therefore, raw SQL is kind of "the bottom half" of Datalog, and to get the upper half you need features like CTEs, VIEWs, etc, and apply them appropriately. You need more concepts to cover both the bottom and top half; Datalog covers them with one concept. Datalog also makes it easy to express things like e.g. queries on graph structures, but again, you don't need extra features like CTEs for this to happen. There are of course lots of tricky bits (e.g. optimization) but the general idea works very well. > Could you share a more specific example or scenario where you have seen Datalog/ Logica outperform a modern SQL setup? Again, Datalog is not about SQL. It's a logic programming language. You need to actually spend time doing logic programming with something like Prolog or Datalog to appreciate the class of things it can do well. It just so happens Datalog is also good for expressing relational programs, which is what you do in SQL. Most of the times I'm doing logic programming I'm actually writing programs, not database queries. Trying to do things like analyze programs to learn facts about them (Souffle Datalog, "can this function ever call this other function in any circumstance?") or something like a declarative program as a decision procedure. For example, I have a prototype Prolog program sitting around that scans a big code repository, figures out all 3rd party dependencies and their licenses, then tries to work out whether they are compatible. It's a bit like Lisp, in the sense that it's a core formulation of a set of ideas that you aren't going to magically adopt without doing it yourself a bunch. I could show you a bunch of logic programs, but without experience all the core ideas are going to be lost and the comparison would be meaningless. For the record, I don't use Logica with SQL, but not because I wouldn't want to. It seems like a good approach. I would use Datalog over SQL happily for my own projects if I could. The reasons I don't use Logica for instance are more technical than anything -- it is a Python library, and I don't use Python.
jyounker 2y ago
The covid analysis seems like a pretty good example: https://colab.research.google.com/github/EvgSkv/logica/blob/main/examples/Logica_example_Covid_Analysis.ipynb https://colab.research.google.com/github/EvgSkv/logica/blob/... A good exercise might be converting it to the corresponding SQL and comparing the two for clarity.

from-nibly 2y ago

Prolog et al is a real brain buster. As in it will break your spirits and build you back up better. I remember in college I was able to build a binary tree with 3 lines of code. And once you write the insert, the delete, search, and others just magically appear. It also frames your thinking about defining what you want rather than how to get it. If you really want to see the power of these kinds of languages look up Einstein's puzzle solved with prolog. The solution just magically comes out by entering the constraints of the puzzle.

rytis 2y ago
I suppose something like this: https://stackoverflow.com/a/8270393 https://stackoverflow.com/a/8270393 ?
surgical_fire 2y ago
I had to use Prolog in college, and while I never saw it in the wild - I at least never stumbled upon a scenario where prolog was the answer - I really enjoyed how I had to change how I looked at a problem in order to solve it in prolog.

burakemir 2y ago

Here is a proof that you can translate non-recursive datalog into relational algebra and vice versa: https://github.com/google/mangle/blob/main/docs/spec_explain_relational_algebra.md https://github.com/google/mangle/blob/main/docs/spec_explain... Since Logica is translated to SQL it should benefit from all the query optimistic goodness that went into the SQL engine that runs the resulting queries. I personally see the disadvantages of SQL in that it is not really modular, you cannot have libraries, tests and such. Disclosure: I wrote Mangle (the link goes to the Mangle repo), another datalog, different way of extending, no SQL translation but an engine library.

aseipp 2y ago
Mangle looks very interesting, thanks for the share. In particular I love your GRPC demo, because it shows a prototype of something I've been thinking about for a long time: what if we did GraphQL, but with Datalog! Maybe we could call it LogiQL :) In particular many people talk a lot about concerns like optimizations across GraphQL plans and how they are expected to behave on underlying tables, but this is something that I think has seen a lot of research in the Datalog realm. And to top it off, even ignoring that, Datalog just feels much more natural to write and read after a bit of practice, I think. (Obviously you need to be in the pure fragment of datalog without recursion, but even then it might be feasible to add those features with termination criteria even if it's just "decrement an internal counter and if it hits zero throw a big error") What do you think the plans for the Rust implementation will be? That's probably the most likely place I'd use it, as I don't really use Go that much.
- burakemir 2y ago
  The Mangle repo has the beginnings of a Rust implementation but it will take some time before it is usable. The go implementation is also still being improved, but I think real DB work with persistent data will happen only in Rust. Bindings to other host languages would also use the Rust implementation. There are no big challenges here it is just work and takes time. The combination of top-down and bottom up logic programming is interesting, especially when one can move work between pre computation and query time. I like that optimizing queries in datalog can be discussed like optimization of programming language but of course the biggest gains in DB come from join order and making use of indices. There is a tension here between declarative and having some control or hints for execution. I haven't yet figured out how one should go about it, and also how to help programmers combine top-down and bottom-up computation. Work in progress! :-)

aseipp 2y ago

> If I understand you correctly, you can easily get the same with ephemeral models in dbt or CTEs generally? You can bolt on any number of 3rd party features or extensions to get some extra thing, that goes for any tool in the world. The point of something like Datalog is that it can express a similar class of relational programs that SQL can, but with a smaller set of core ideas. "Do more with less." > I guess, but how can you guarantee correctness with flexibility involved? How do you guarantee the correctness of anything? How do you know any SQL query you write is correct? Well, as the author, you typically have a good idea. The point of being compositional is that it's easier to stick together arbitrary things defined in Datalog, and have the resulting thing work smoothly. Going back to the previous example, you can define any two "tables" and then just derive a third "table" from these, using language features that you already use -- to define relationships between rows. Datalog can define relations between rules (tables) and between facts (rows), all with a single syntactic/semantic concept. While SQL can only by default express relations between rows. Therefore, raw SQL is kind of "the bottom half" of Datalog, and to get the upper half you need features like CTEs, VIEWs, etc, and apply them appropriately. You need more concepts to cover both the bottom and top half; Datalog covers them with one concept. Datalog also makes it easy to express things like e.g. queries on graph structures, but again, you don't need extra features like CTEs for this to happen. There are of course lots of tricky bits (e.g. optimization) but the general idea works very well. > Could you share a more specific example or scenario where you have seen Datalog/ Logica outperform a modern SQL setup? Again, Datalog is not about SQL. It's a logic programming language. You need to actually spend time doing logic programming with something like Prolog or Datalog to appreciate the class of things it can do well. It just so happens Datalog is also good for expressing relational programs, which is what you do in SQL. Most of the times I'm doing logic programming I'm actually writing programs, not database queries. Trying to do things like analyze programs to learn facts about them (Souffle Datalog, "can this function ever call this other function in any circumstance?") or something like a declarative program as a decision procedure. For example, I have a prototype Prolog program sitting around that scans a big code repository, figures out all 3rd party dependencies and their licenses, then tries to work out whether they are compatible. It's a bit like Lisp, in the sense that it's a core formulation of a set of ideas that you aren't going to magically adopt without doing it yourself a bunch. I could show you a bunch of logic programs, but without experience all the core ideas are going to be lost and the comparison would be meaningless. For the record, I don't use Logica with SQL, but not because I wouldn't want to. It seems like a good approach. I would use Datalog over SQL happily for my own projects if I could. The reasons I don't use Logica for instance are more technical than anything -- it is a Python library, and I don't use Python.

jyounker 2y ago

The covid analysis seems like a pretty good example: https://colab.research.google.com/github/EvgSkv/logica/blob/main/examples/Logica_example_Covid_Analysis.ipynb https://colab.research.google.com/github/EvgSkv/logica/blob/... A good exercise might be converting it to the corresponding SQL and comparing the two for clarity.