12 ms·

Introduction to Datalog

362 points by jgrodziski 4y ago

ianpurton 4y ago
So I struggled with this. I guess the intention is to be better than SQL but then I was left with "under which circumstances?". With that question in mind I didn't feel the article addressed the issue. The author might do better to think in terms of "what burning problem are we trying to fix and how did we fix it".
- YeGoblynQueenne 4y ago
  >> The author might do better to think in terms of "what burning problem are we trying to fix and how did we fix it". I have no idea who first came up with the name "datalog", and what exactly was their motivation (I've looked, but couldn't find the original reference to datalog), but the burning problem that datalog does fix is Prolog's semi-decidability, or in other words, its tendency to enter infinite recursions. One half of the reason for that is Prolog's "compound terms" (known as functions, in First-Order Logic terminology) and the other half is Prolog's use of depth-first search for evaluation. Functions, along with logic variables, mean that a predicate's Herbrand base (its set of logical atoms) can be expanded forever: if f(x) is a term, f(f(x)) is a term, f(f(f(x))) is a term... and so on, ad infinitum. The depth-first search evaluation just gets stuck in left-recursions when the first body literal of a clause is a recursive literal: p(x):- p(x), q(x) loops forever, in Prolog with depth-first search. Datalog is Prolog without functions other than constants, and it can be evaluated "bottom up", both of which overcome Prolog's semi-decidability (but eliminate its completeness). But of course this is not a very clear motivation for its use as a database language, only its use as an alternative to Prolog. More to the point, there are many datalog variants. Some that accept negation, some that do not, some that allow recursion, some that do not, and so on. There's a lot of information on different datalogs, seen from the point of view of database research in this free ebook on databases: http://webdam.inria.fr/Alice/ http://webdam.inria.fr/Alice/ See chapters 12 - 15. And see these lecture notes for a more logic-programm-y discussion of fixpoint semantics of definite logic programs: https://www.doc.ic.ac.uk/~mjs/teaching/KnowledgeRep491/Fixpoint_Definite_491-2x1.pdf https://www.doc.ic.ac.uk/~mjs/teaching/KnowledgeRep491/Fixpo...
  - falsissime 4y ago
    > but the burning problem that datalog does fix is Prolog's semi-decidability, or in other words, its tendency to enter infinite recursions. Prolog's built-in search is not semidecidable. https://en.wikipedia.org/wiki/Decidability_(logic)#Semidecidability https://en.wikipedia.org/wiki/Decidability_(logic)#Semidecid... As you observe, Prolog gets stuck in left-recursions. But iterative deepening for Prolog is semidecidable (among other fair methods). This indeed is restricted to the pure, monotonic subset of (full) Prolog together with all pure, monotonic extensions.
    - YeGoblynQueenne 4y ago
      >> As you observe, Prolog gets stuck in left-recursions. But iterative deepening for Prolog is semidecidable (among other fair methods). This indeed is restricted to the pure, monotonic subset of (full) Prolog together with all pure, monotonic extensions. Thanks, yes, I might be fudging terminology a bit. Generally, when I talk about Prolog I mean definite programs (sets of definite clauses and Horn goals, the latter usually called "definite goals") under SLD-resolution. That's more or less what is usually meant by "pure" Prolog: definite programs, without not/1 implementing Negation-As-Failure, without the cut, and without side effects; and executed by giving an initial Horn goal as a "query". Entailment between definite clauses and satisfiability of definite programs is undecidable. By "semi-decidable" I mean that an SLD-refutation proof will terminate if there is a branch of the proof that ends in □ and is of finite length. If such a branch does not exist, a proof will "loop" forever. That's regardless of whether the branch succeeds or fails, which is a bit of a fudge, but, in practice, there is no other way to decide the success or failure of a proof than to search all its branches. Left-recursion is one way in which Prolog, with depth-first search, generates branches of infinite length, but you can get those with right-recursion also. There are restrictions of Prolog, like SLG-resolution (similar to DFS with iterative deepening) that don't loop on left-recursions but the general case remains undecidable. Fortunately, there seem to be at least as many finite proofs as there are infinite ones, or in any case I have never encountered a Prolog program that looped inintially and that couldn't be rewritten to terminate, at least when called in the way it was meant to be called. And that's also a bit of a fudge.
- bambataa 4y ago
  I had the exact same reaction. I did a bit of reading about Datalog and I get that it’s useful in things like static analysis where you get the fixed point analysis for free, but apart from that I am not really sure why I’d use it.
  - maweki 4y ago
    The main reasons I'd say are the very intuitive JOINs and queries. Like you give some partial relation and you get every "fit". So as long as you mostly do equalities in JOIN and WHERE, the queries will be very intuitive and obvious to the layperson. If you work with VIEWs or CTEs a lot, then Datalog is your friend, as every derived relationship is just a VIEW. If you like to encapsulate and reuse queries, then Datalog is ideal. We had plenty discussions on HN about SQL code reuse. Not an issue in Datalog. And lastly, there are many ways to compile Datalog programs to "executables" that take maybe a few CSV-files as input and give you the results. I'm not saying that this would be always faster than loading the CSVs into SQLite and running SQL against the data, but a lot of work has been done on Datalog query optimization and compilers can emit very efficient code.
    - samuell 4y ago
      > If you like to encapsulate and reuse queries This is what I always thought was the main benefit. You are able to much easier raise your level of abstraction, by building up a vocabulary of definitions, and model your further queries (or definitions) using them.
- sirwhinesalot 4y ago
  That's a good point, to be honest I'm not sure there's a particularly good answer to that beyond "it's a much better underlying model". SQL is an ugly bastardization of the relational model, whereas Datalog is a clean application of the logical model to the relational setting. One thing that's really nice about Datalog is that it puts the focus on modelling relations between data instead of "tables" which often end up with a jumble of data and then need to be normalized to work properly. It pushes you into good structures by default, which is a really good property to have. It's also much much simpler than SQL, no need to think in terms of inner joins and outer joins and whatever else, it is all relations. It also easily produces derived data from existing data, without needing any kind of procedural process, and is highly composable. It's not so much that it's solving a burning problem than SQL, it's just better than SQL... (note: the clojure syntax used in that page is much more confusing than the native Datalog syntax IMO)
  - zh217 4y ago
    Another problem with Clojure-based Datalog is performance. Yet another is you are pretty much tied to the Clojure ecosystem. And I don't really like the Clojure-fused Datalog syntax either. These pains spurred me to write my own: https://docs.cozodb.org/en/latest/ https://docs.cozodb.org/en/latest/ (FOSS), and so far I'm satisified with my own work.
    - arohner 4y ago
      What is the performance problem?
- yaantc 4y ago
  In 2021 Google introduced Logica, a Datalog variant. In their introduction blog they contrast it with SQL, so this may answer your question: https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html https://opensource.googleblog.com/2021/04/logica-organizing-... In short, it's easier to compose (and decompose), which helps with complex queries that can be assembled from simpler, independently tested parts. No personal experience in this, I just remembered that blog contrasting Datalog and SQL.
- ekidd 4y ago
  > I guess the intention is to be better than SQL but then I was left with "under which circumstances?" Excellent question. Two of the most common use cases for databases are "transactional processing" (manipulating small numbers of rows in real time) and "analytical processing" (querying enormous numbers of rows, typically in a read-only fashion). SQL is generally fine for transactional workloads. But analytical queries sometimes involve multi-page queries, with lots of JOINs and CTEs. And these queries are often automatically generated. And once you start writing actual multi-page "programs" in SQL, you may decide that it's a fairly clunky and miserable programming language. What Datalog typically buys you is a way to cleanly decompose large queries into "subroutines." And it offers a simpler syntax for many kinds of complex JOINs. Unfortunately, there isn't really a standard dialect of Datalog, or even a particular dialect with mainstream traction. So choosing Datalog is a bit of a tradeoff: does it buy you enough, for your use case, that it's worth being a bit outside the mainstream? Maybe! But I'd love to see something like Logica gain more traction: https://logica.dev/ https://logica.dev/
- anon291 4y ago
  Sql doesn't compose at all. You can't take two sql programs and combine them with a single operator and get a new program. In datalog you can. Either by using the comma or semicolon operator.
  - crabbone 4y ago
    You are thinking about the declarative part. SQL/STP, the standard that's the basis for definition of the imperative part of SQL (stored procedures) can compose in a similar way (eg. you can call a stored procedure from a stored procedure), but the standard doesn't go far enough, and so every practical database has its own extensions which deters people from using stored procedures because they'd end up with non-portable code.
    - anon291 4y ago
      stored procedures, being procedural, are the antithesis of a declarative database library. I'm talking about this: Query 1: SELECT a, b, c FROM table1 JOIN table2 ... Query 2: SELECT d, e, f FROM (SUBSELECT ... ) AS table3 JOIN table4 ... Now, join those queries together. How? Well you have to analyze the join clauses in from and then combine those together, then rewrite the SELECTION. Okay, great... not so hard. Now use Query 2 in a recursive CTE for Query 1. Oh goodness... much harder. Or what about: Query 1: SELECT a, b, c FROM table1 JOIN table2 ... Query 2: SELECT d, e, f FROM (SUBSELECT ... ) AS table3 JOIN table4 ... ORDER BY table4.column LIMIT 10 Now join 1 and 2 together... the syntax is much much much different. Some SQL libraries in Haskell, like Beam, Opaleye, Squeal, etc, demonstrate the kind of composability I mean. In particular, all three of these offer a monadic interface where arbitrary queries can be joined using well-known relational operators (relational cross joins form a monad). I'm not sure of Opaleye and Squeal, but Beam does it's best to produce a human-comprehensible query. Moreover, the first queries given above are 'easy' in the sense that the portion after the `FROM` in a regular SQL SELECT does actually compose well, since the JOIN operators are basically a direct translation of the underlying relational algebra. However, they start to fail when you want to join up aggregations and the like. While true that aggregations and ordering are not strictly relational, it is of great use to be able to compute with such things. Again, the libraries above demonstrate the composability. For example, the beam library offers this: https://haskell-beam.github.io/beam/user-guide/queries/aggregates/ https://haskell-beam.github.io/beam/user-guide/queries/aggre... Notice how we are able (in the second to last query), simply combine the query `all_ (track chinookDb)` with the complex aggregate expression. In this DSL, the entire aggregate could be floated out. In particular, notice how the queries for either the first `all_` or the second complex aggregate would look much different on their own. That's what I mean by SQL doesn't compose. Of course all these DSLs have the problem that they're trying to work with an existing language SQL that is not nice and orthogonal, and so they all have their varying deficiencies. Datalog mostly fixes these issues and provides a nice model.
- felixyz 4y ago
  > then I was left with "under which circumstances?" Foremost, Datalog has great compositionality, which is one of SQL:s big weaknesses (it was designed with completely different goals). Also, things like recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions.
  - refset 4y ago
    > Datalog has great compositionality, which is one of SQL's big weaknesses There's a good write-up on this aspect (amongst others) here: https://www.scattered-thoughts.net/writing/against-sql/ https://www.scattered-thoughts.net/writing/against-sql/ > recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions And similarly: https://github.com/frankmcsherry/blog/blob/master/posts/2022-12-25.md https://github.com/frankmcsherry/blog/blob/master/posts/2022...
    - felixyz 4y ago
      Of all the anti-SQL screeds out there, this is my favorite as well. And Frank McSherry... well, everything he writes is gold.
- dgb23 4y ago
  One of the biggest advantages is re-use, extensibility and being robust to change, because you define things bottom up. Think of aggregates as "hashmaps with typed keys" rather than "structs with places in them". Aggregates are implicit groupings, not explicit slots. This wards you from risk of change and speeds up your feedback loop of making changes and experimenting with ideas. Plus you can compose new aggregate variants out of the same primitives (attributes) which is a fairly typical use-case. Another is query composition. I'm more familiar with SQL and I'm somewhat comfortable with writing nested queries. But we all know how both query builders and similar are often hard to re-use and compose. With these datalog dialects, composition, refactoring and extraction of logical units is much more straight forward, even as a beginner. Queries are "flat", and consist clauses that can be moved around and composed. Think of having more associative and commutative operations. On top of these, the mentioned technologies (in the article) are temporal and support time travel out of the box. That's something you can do on top of SQL of course but it's quite powerful to have it in-built as a primary feature. Whenever you have an application that does things like (non-exclusive), reporting, undo, revisions etc. You might want that.
- crabbone 4y ago
  I'm not the author, and I don't know how author would answer this question, but some things are really on the surface: 1. Modularity. In Datalog it's very easy to name a common (sub-)query and reuse it in multiple queries. You can have stored procedures / functions in SQL, but the syntax is very different implementation to implementation and users are typically afraid of using the feature (the SQL/PSM does not go far enough to define what users would normally want). Also, SQL/PSM is an imperative extension of otherwise declarative language. The two just don't work well together. 2. Structures other than primitives and tables (you get lists, vectors, hash-maps and sets, but you can also make your own). A lot of practical SQL extensions also offer some of these structures, but they aren't in the standard. 3. Operations on primitive types fall into the same category / work in the same way as operations on complex types. I.e. select / update / insert / delete in SQL only apply to tables, but strings or numbers don't work in the same way. It's more uniform in Datalog (not 100%, but still better).
- conor-23 4y ago
  A researchy perspective: Datalog was invented to extend relational algebra with recursion. Since it started out as an academic tool, people have been studying recursion-specific optimizations you can do for decades so it is extremely well suited to recursive use-cases e.g. iterative graph algorithms. Using Datalog for network algorithms won the thesis award in databases almost 20 years ago https://boonloo.cis.upenn.edu/papers/boon_interview.pdf https://boonloo.cis.upenn.edu/papers/boon_interview.pdf .
  - tejtm 4y ago
    This is the answer I subscribe to. CTEs and recursive CTEs are SQL's answer to a limitation of plain relational algebra; no loops. CTEs are a great and most welcome addition to SQL but they are a bolt-on patch as compared with Datalog where it is a core feature.
  - felixyz 4y ago
    > Datalog was invented to extend relational algebra with recursion. I'm not sure that is exactly right. Do you have a reference? (Not trying to put you on the spot, I'm just curious to learn the history!)
    - YeGoblynQueenne 4y ago
      Yeah, if the OP can give a reference I'd be very interested, too. I've searched for the "original" reference to datalog because I wanted to cite it, but I couldn't find anything like that. I have a sneaking suspicion that "function-free Prolog" is as old as ordinary Prolog, and "datalog", as an idea separate to Prolog and used as a database language, was born in the database community, but like the OP I have no reference to this.
    - refset 4y ago
      Agreed, my understanding is that Datalog has a distinct (though related) lineage that directly emerged from Prolog (i.e. logic programming, not relational algebra / database theory) - skimming the introduction of "Horn Clauses and the Fixpoint Query Hierarchy (1982)" seems to confirm this: https://dl.acm.org/doi/pdf/10.1145/588111.588137 https://dl.acm.org/doi/pdf/10.1145/588111.588137 Edit: this presentation describes things differently but it doesn't sound quite right to me "Chandra and Harel - 1982 Studied the expressive power of logic programs without function symbols on relational databases" https://www.dbai.tuwien.ac.at/datalog2.0/slides/Kolaitis.pdf https://www.dbai.tuwien.ac.at/datalog2.0/slides/Kolaitis.pdf
- mmcdermott 4y ago
  > I guess the intention is to be better than SQL but then I was left with "under which circumstances?". The way I see it, SQL's great strength as a query language is selection and projection whereas Datalog's great strength is inference. It takes a shift in mindset to take advantage of inference. I've started using Datalog when analyzing unfamiliar codebases. So I might set up something like: writes_to(microservice1, db1). writes_to(microservice2, microservice1). writes_to(microservice3, microservice2). depends_on(X, Y) :- writes_to(X, Z), depends_on(Z, Y). depends_on(X, Y) :- writes_to(X, Y). Allowing you to query something like: depends_on(microservice3, X) and get back db1, microservice1 and microservice2. Of course, you can ultimately do this in SQL as well, but it's far more compact in Datalog. Which brings me to the second half of the motivation - the principal advantage to the logical database model is the open world assumption. The relational model in practice tends to be fairly buttoned down - data is expected to suit a particular schema (which has its own advantages as a transactional system). This makes it easy to extend a logical database and ask more questions without changing the fact formats already specified. Of course, I can ultimately do all the things that feel natural in Datalog in SQL. I can work through queries like the one above by building out the data model and writing recursive CTEs. It's about strengths, not possibilities which I suppose brings me back to why SQL 'won'. A disproportionate amount of code is written for transactional line-of-business systems. It's really not hard to see why the validation layer and transactional focus would win in those areas.
pavlov 4y ago
As mentioned in the article, Datomic is a database that uses Datalog as its query language: https://docs.datomic.com/on-prem/query/query.html#why-datalog https://docs.datomic.com/on-prem/query/query.html#why-datalo... (Some ten years ago worked at a startup that used Datomic. It seemed to work great, although the only queries I ever needed to add to the system were simple copy-paste hacks of existing ones, so I never got to dive into Datalog.)
- dmitriid 4y ago
  Datascript is the open source analog for Clojure, ClojureScript and JS: https://github.com/tonsky/datascript https://github.com/tonsky/datascript
  - simongray 4y ago
    There are many open source alternatives in Clojure using this query language: https://github.com/simongray/clojure-graph-resources#datalog https://github.com/simongray/clojure-graph-resources#datalog
  - mpenet 4y ago
    'ish. datahike would be the closest to datomic in terms of features/implementation (support for as-of, transactor etc). Then in terms of maturity I think the choice is between xtdb and datascript, both are very solid/maintained but they are also vastly different.
tannhaeuser 4y ago
The language discussed in TFA appears to be Datomic's proprietary Clojure DSL, but has nothing to do with Datalog/Prolog.
- dimitar 4y ago
  It is a datalog dialect, and there a multiple open-source implementations: https://clojupedia.org/#/page/Datalog https://clojupedia.org/#/page/Datalog
thiago_fm 4y ago
The problem with Datalog, and Clojure in general are the licenses. Terrible licenses. Everything is about Rich Hickey. Apache 1.0. Now that Nubank basically owns it and there's very little progress or activity as of late, I don't see why one would chose to use Clojure, Datalog etc. Also, a lot of functional programming concepts has been since added to big programming languages like Javascript and hell, even Java has lambdas now. I'm guessing that also hardcore FP people have moved on to Haskell. The ones that like LISP to Racket... and only people tied to the JVM in legacy projects are with Clojure.
- maweki 4y ago
  What are the licensing issues with First-order Horn clause logic? Datalog is not a licensed product or software, the same way Answer Set Programming isn't.
- casion 4y ago
  > very little progress or activity as of late There's been equally frequent releases and updates, I'm not sure how you came to this conclusion? The best I can think of is that you're confusing core language updates with other tech. Clojure is structured differently than other languages, and core language/library updates are (and always have been) relatively rare while the surrounding ecosystem/tooling provided by the Clojure team is active.
- armincerf 4y ago
  XTDB is MIT and has clients for Java or HTTP if you don't want to write Clojure. Not every Datalog implementation is 'about Rich Hickey'
- cmrdporcupine 4y ago
  Datalog != Datomic. Datomic ∈ Datalog
  - refset 4y ago
    It's true that this SPARQL-inspired view of Datalog as a triplestore query language is quite a narrow interpretation compared to something closer to the academic Prolog roots like https://souffle-lang.github.io/ https://souffle-lang.github.io/ - what do you feel are the most important differences?
wslh 4y ago
In the last few months the mention of Datalog has increased, I wondered how it differed from graph databases and found a clear answer in SO [1]. I am not an incumbent but found graph databases and clause approaches interesting. [1] https://stackoverflow.com/questions/29192927/a-graph-db-vs-a-prolog-or-minikanren https://stackoverflow.com/questions/29192927/a-graph-db-vs-a... (2015)
- noduerme 4y ago
  That's a really neat example of something I'm not familiar with. Going up a tree from child to parent is often the heaviest part of dealing with regular datasets, and usually requires a mix of queries and application logic. The idea of flattening the data along some pattern like that is of course always possible in a relational db, but it's not usually efficient, especially not for heavy writing. Lateral joins and window partitions can help. But this seems like an interesting approach to removing the app code completely.
- refset 4y ago
  XTDB, which is mentioned in the post, is subtly different from the other Clojure-based Datalog systems in this respect, because its Datalog engine executes in terms of multi-way joins using a "Worst-Case Optimal Join" implementation that is ideal for graph processing (vs. a tree of binary hash joins). Therefore, based on statistics and query planning heuristics, it will often perform graph pattern matching before resolving the logic/horn clauses. (source: I work on the XTDB team)
  - eternalban 4y ago
    Interesting architecture: https://raw.githubusercontent.com/xtdb/xtdb/master/docs/concepts/modules/ROOT/images/xtdb-node-1.svg https://raw.githubusercontent.com/xtdb/xtdb/master/docs/conc... Btw, is that 'RocksDB or ?' for the local store current or other storage engines can get plugged in? p.s. this is datomic's architecture for comparison. https://docs.datomic.com/on-prem/images/clientarch_orig.svg https://docs.datomic.com/on-prem/images/clientarch_orig.svg
    - 4y ago
dataengineer56 4y ago
This is a cool concept but I'm not sure I'd fancy upskilling a team of SQL analysts to use this.
- cmrdporcupine 4y ago
  Many SQL analysts are wasting their skills performing mental and syntactical gymnastics to get around the limitations of SQL in order to grasp at the actual conceptual elegance that lays underneath it. Most people who are writing SQL for a living already understand at least some part of what makes the relational model powerful. But SQL is relatively poor tool for accessing it. I personally don't find the Sexpr-based syntax of Datomic's variant of Datalog all that useful here, and yeah, maybe someone working in SQL for a living would struggle at first with that syntax. But it's not intrinsic to the model itself. Have a waltz through my employer's documentation and see what you think: https://docs.relational.ai/rel/primer/overview https://docs.relational.ai/rel/primer/overview I think it's quite understandable (if a bit terse) and many people doing SQL for a living would appreciate the ability to better compose and structure things in this way, not to mention the ability to handle transitive / recursive relationships in a less awkward way.
eunos 4y ago
Huh cool, I didnt realize it come from that Michelin.
- eddieroger 4y ago
  I remember being surprised once that Michelin, like the star, was the same as Michelin, like the tire. It's really cool to see a company move beyond their core competency more than once in a meaningful way, even if the first time was to sell more tires. They have a really interesting blog that this is just a single article from.
  - pbronez 4y ago
    I believe they started the star ratings as a way to encourage people to drive more. They saw it as part of “Travel”.
    - julienchastang 4y ago
      Yes, absolutely, as are the eponymous "Guide Michelin" (Guide vert) that conveniently fit in the glove compartment of your automobile. More people drive, more they consume tires.
k4st 4y ago
I created a datalog engine a few years back called Dr. Lojekyll: https://www.petergoodman.me/docs/dr-lojekyll.pdf https://www.petergoodman.me/docs/dr-lojekyll.pdf It was pretty cool; you could stream in new facts to it over time and it would incrementally and differentially update itself. The key idea was that I wanted the introduction of ground facts to be messages that the database reads (e.g. off of a message bus), and I wanted the database to be able to publish its conclusions onto the same or other message buses. I also wanted to be able to delete ground facts, which meant it could publish withdrawals of the prior-published conclusions. A lot of it was inspired by Frank McSherry's work, although I didn't use timely or differential dataflow. In retrospect I probably should have! This particular system isn't used anymore because we made a classic monotonicity mistake by making it the brain of a distributed system, and then having it publish and receive messages with a bunch of microservices. The internal consistency model of the datalog engine didn't extend out to the microservices, and the possibility of feedback loops in the system meant that the whole thing could lie to itself and diverge uncontrollably! Despite this particular application of the engine being a failure, the engine itself worked quite well and I hope to one day return to datalog. I think what a lot of people miss with datalog, and what becomes apparent as you use it more, is just how unpredictable many engines can be with the execution behavior of rules. This is the same problem that you have with a database, where the query planner makes a bad choice or where you lack an index, and so performance is bad. But with datalog, the composition of rules that comes so naturally also tends to compound this issue, resulting in time spent trying to chase down weird performance things and doing spooky re-ordering of your clause bodies to try to appease whatever choices the engine makes.
samuell 4y ago
A bit related, just stumbled upon Flix, a functional JVM language with Datalog contraints and (somewhat?) Go-like concurrency: https://flix.dev https://flix.dev HN Thread from 8 months ago: https://news.ycombinator.com/item?id=31448889 https://news.ycombinator.com/item?id=31448889
- refset 4y ago
  Flix definitely looks interesting! For comparison, I ported the "Datalog Enriched with Lattice Semantics" example from that homepage to XTDB's (Clojure) Datalog after I saw it posted on HN originally: https://gist.github.com/refset/21b3fc1dec9a6928943073809e13356d https://gist.github.com/refset/21b3fc1dec9a6928943073809e133...
muattiyah 4y ago
ICYMI, there's an excellent interactive introduction to `datalog` that's referenced in the article's references.[0] Last time I used `datalog` was years ago, I was developing an internal interactive tool that was used to compare different approaches to solving a certain problem at my employer. I used `datascript`[1] by way of clojurescript to store all experiment data and then interrogated the `datascript` DB via `datalog`. This is something I always remember fondly. [0] https://www.learndatalogtoday.org/ https://www.learndatalogtoday.org/ [1] https://github.com/tonsky/datascript https://github.com/tonsky/datascript
anon291 4y ago
Whatever language this is... This is not datalog. This looks like a particular implementation of datalog in closure. Actual datalog looks like prolog.
- cmrdporcupine 4y ago
  In reality, people are using "datalog" for a genre of datastore concepts based around horne clauses, or, basically relations + implicit joins. Datalog as a subset or dialect of prolog is only one variant of this. And Datomic has made an sexpr-syntaxed variant built around binary relations popular. To the point where some people in this thread can't seem to tell the two apart. I am more interested in the general category of relational data model + logic programming than I am in any purity about Datalog in particular. In particular I'm very excited by "data/knowledge + behaviour sitting in a tree, k-i-s-s-i-n-g"
  - bogomipz 4y ago
    Thanks. Is Datomic also a dialect of Prolog then?
    - cmrdporcupine 4y ago
      No, and I wouldn't call Datalog a dialect of Prolog either necessarily. It's more like, classic Datalog syntax looks Prolog-ish (similar syntax, and is kind of a subset of Prolog). And you can do Datalog-type stuff in Prolog. But from what I've read of Datomic it's got a totally different syntax, but similar semantics to Datalog .. but not Prolog... Basically, Prolog can do more than Datalog. Datalog is like, some subset of concepts of Prolog, but specialized for relational data queries, so it can achieve some optimizations and focus on data retrieval only. Is that confusing enough?
      - dragonwriter 4y ago
        > No, and I wouldn’t call Datalog a dialect of Prolog either necessarily. It’s more like, classic Datalog syntax looks Prolog-ish (similar syntax, and is kind of a subset of Prolog). Datalog originated specifically as a restricted subset of Prolog, which is why the “classic” syntax looks prolog-ish. > Basically, Prolog can do more than Datalog. Datalog is like, some subset of concepts of Prolog, but specialized for relational data queries, so it can achieve some optimizations and focus on data retrieval only. Datalog is a purely declarative, non-Turing complete subset of Prolog, removing the imperative features and assuring termination. This gives up lots of power, obviously, but it also, well, provides a termination guarantee. Removing imperative features (cut) from Prolog also means that Datalog implementations can use different (or multiple, switchable) strategies, while differing in what kinds of data sets and queries they perform well on, not correctness.
i_am_toaster 4y ago
Maybe it’s just me but I find the SQL much easier to read in all the examples given.
grose 4y ago
Datalog is great for representing authorization rules. Check out Biscuits, which are auth tokens with Datalog embedded in them. This article is what made it 'click' for me: https://www.clever-cloud.com/blog/engineering/2021/04/15/biscuit-tutorial/ https://www.clever-cloud.com/blog/engineering/2021/04/15/bis... I actually thought that Datalog was so cool that I went to learn Prolog and it completely changed the way I think about programming. Highly recommend trying out logic programming if you haven't before.
- burakemir 4y ago
  Agree but also want to point out that people usually have a narrow view on "logic programming". Datalog can also be understood with out the top-down evaluation / resolution that is typically associated with prolog, which is why it is known to database researchers and in finite model theory. Prolog is great, but bottom up techniques to evaluate datalog are awesome, too, and would arguably also qualify as logic programming. It is rare to see this acknowledged.
  - YeGoblynQueenne 4y ago
    Acknowledged, by whom? As far as I know people in the logic programming community have no trouble recognising datalog as a logic programming language, be it evaluated bottom-up or not (you can still evaluate a datalog program top-down, by resolution, as if it were a Prolog program without "compound" terms) (a.k.a. functions). Indeed, I get the feeling that the primacy of Prolog as the logic programming language has waned. If you look at back issues of ICLP* proceedings you'll find plenty of work that is nothing to do with resolution- namely, Answer Set Programming, which is wildly popular. I tend to think it's mainly the database community that kind of ignores the logic-programming nature of datalog. Oh and btw, when I talk about datalog, I mean definite clauses without functions, not the aberrant syntax in the article above. I'll never understand why people do that. ________________ * ICLP is the International Conference on Logic Programming.
z5h 4y ago
I’ve been using Prolog a bunch recently, and also embedded and extended MicroKanren in a project. Something I came to appreciate was that Prolog’s depth-first search, and Kanren’s lazy stream approach are good with memory even when generating/searching through infinite solutions. It is my understanding that Datalog, on the other hand, will iteratively expand a set of data. Isn’t this a problem?
- YeGoblynQueenne 4y ago
  "Iteratively expand a set of data"? I'm not sure what you mean here. I think you are probably talking about the "bottom up" evaluation strategy of datalog, right? That's where datalog is evaluated by a so-called TP-operator, which derives the set of all logical consequences a datalog program by calculating its least fixed point. That's the same as the Least Herbrand Model (LHM) of the program, or, in other words, the set of atoms entailed by the program (atoms in the logical sense, of atomic formulae, not in the Prolog sense of constants). That's the same thing that Prolog does, calculate the LHM of a logic program, but the difference is that datalog programs have finite LHMs, because they don't have functions that can be self-instantiated for ever ( f(x), f(f(x)), f(f(f(x))), ... ) and the bottom-up evaluation, that goes from the "body" to the "head" of a clause, avoids infinite left-recursions. Prolog, evaluated top-down (clauses are "picked apart" head-first) can get stuck in infinite left-recursions, so datalog's finiteness, and its decidability under TP, is a big gain in efficiency, as anyone who has had to kill a Prolog console session because of an infinite loop will know. Also, it is not widely recognised but I am the author of the dumbest and most inefficient TP Operator implementation in existence. Obviously I hang my head in shame and will not link to my code. I understand however that there are optimisations that one can perform that make bottom-up execution efficient, and even quite fast. Unfortunately, I don't know what they are :P Note that Prolog can also be evaluated without fear of left-recursions, by SLG-Resolution (a.k.a. "tabling", a.k.a. memoization) but there is still the danger of infinite right recursions. Prolog is semi-decidable, because it is Turing-complete. Datalog is decidable, sacrificing completeness for, well, efficiency. So, in short, it's not a problem if you consider the alternative, but of course there are trade-offs, always. It's like growing old, vs. dying young. (I hope all this is not completely irrelevant to your question).

362 points by jgrodziski 4y ago

So I struggled with this. I guess the intention is to be better than SQL but then I was left with "under which circumstances?". With that question in mind I didn't feel the article addressed the issue. The author might do better to think in terms of "what burning problem are we trying to fix and how did we fix it".

YeGoblynQueenne 4y ago
>> The author might do better to think in terms of "what burning problem are we trying to fix and how did we fix it". I have no idea who first came up with the name "datalog", and what exactly was their motivation (I've looked, but couldn't find the original reference to datalog), but the burning problem that datalog does fix is Prolog's semi-decidability, or in other words, its tendency to enter infinite recursions. One half of the reason for that is Prolog's "compound terms" (known as functions, in First-Order Logic terminology) and the other half is Prolog's use of depth-first search for evaluation. Functions, along with logic variables, mean that a predicate's Herbrand base (its set of logical atoms) can be expanded forever: if f(x) is a term, f(f(x)) is a term, f(f(f(x))) is a term... and so on, ad infinitum. The depth-first search evaluation just gets stuck in left-recursions when the first body literal of a clause is a recursive literal: p(x):- p(x), q(x) loops forever, in Prolog with depth-first search. Datalog is Prolog without functions other than constants, and it can be evaluated "bottom up", both of which overcome Prolog's semi-decidability (but eliminate its completeness). But of course this is not a very clear motivation for its use as a database language, only its use as an alternative to Prolog. More to the point, there are many datalog variants. Some that accept negation, some that do not, some that allow recursion, some that do not, and so on. There's a lot of information on different datalogs, seen from the point of view of database research in this free ebook on databases: http://webdam.inria.fr/Alice/ http://webdam.inria.fr/Alice/ See chapters 12 - 15. And see these lecture notes for a more logic-programm-y discussion of fixpoint semantics of definite logic programs: https://www.doc.ic.ac.uk/~mjs/teaching/KnowledgeRep491/Fixpoint_Definite_491-2x1.pdf https://www.doc.ic.ac.uk/~mjs/teaching/KnowledgeRep491/Fixpo...
- falsissime 4y ago
  > but the burning problem that datalog does fix is Prolog's semi-decidability, or in other words, its tendency to enter infinite recursions. Prolog's built-in search is not semidecidable. https://en.wikipedia.org/wiki/Decidability_(logic)#Semidecidability https://en.wikipedia.org/wiki/Decidability_(logic)#Semidecid... As you observe, Prolog gets stuck in left-recursions. But iterative deepening for Prolog is semidecidable (among other fair methods). This indeed is restricted to the pure, monotonic subset of (full) Prolog together with all pure, monotonic extensions.
  - YeGoblynQueenne 4y ago
    >> As you observe, Prolog gets stuck in left-recursions. But iterative deepening for Prolog is semidecidable (among other fair methods). This indeed is restricted to the pure, monotonic subset of (full) Prolog together with all pure, monotonic extensions. Thanks, yes, I might be fudging terminology a bit. Generally, when I talk about Prolog I mean definite programs (sets of definite clauses and Horn goals, the latter usually called "definite goals") under SLD-resolution. That's more or less what is usually meant by "pure" Prolog: definite programs, without not/1 implementing Negation-As-Failure, without the cut, and without side effects; and executed by giving an initial Horn goal as a "query". Entailment between definite clauses and satisfiability of definite programs is undecidable. By "semi-decidable" I mean that an SLD-refutation proof will terminate if there is a branch of the proof that ends in □ and is of finite length. If such a branch does not exist, a proof will "loop" forever. That's regardless of whether the branch succeeds or fails, which is a bit of a fudge, but, in practice, there is no other way to decide the success or failure of a proof than to search all its branches. Left-recursion is one way in which Prolog, with depth-first search, generates branches of infinite length, but you can get those with right-recursion also. There are restrictions of Prolog, like SLG-resolution (similar to DFS with iterative deepening) that don't loop on left-recursions but the general case remains undecidable. Fortunately, there seem to be at least as many finite proofs as there are infinite ones, or in any case I have never encountered a Prolog program that looped inintially and that couldn't be rewritten to terminate, at least when called in the way it was meant to be called. And that's also a bit of a fudge.
bambataa 4y ago
I had the exact same reaction. I did a bit of reading about Datalog and I get that it’s useful in things like static analysis where you get the fixed point analysis for free, but apart from that I am not really sure why I’d use it.
- maweki 4y ago
  The main reasons I'd say are the very intuitive JOINs and queries. Like you give some partial relation and you get every "fit". So as long as you mostly do equalities in JOIN and WHERE, the queries will be very intuitive and obvious to the layperson. If you work with VIEWs or CTEs a lot, then Datalog is your friend, as every derived relationship is just a VIEW. If you like to encapsulate and reuse queries, then Datalog is ideal. We had plenty discussions on HN about SQL code reuse. Not an issue in Datalog. And lastly, there are many ways to compile Datalog programs to "executables" that take maybe a few CSV-files as input and give you the results. I'm not saying that this would be always faster than loading the CSVs into SQLite and running SQL against the data, but a lot of work has been done on Datalog query optimization and compilers can emit very efficient code.
  - samuell 4y ago
    > If you like to encapsulate and reuse queries This is what I always thought was the main benefit. You are able to much easier raise your level of abstraction, by building up a vocabulary of definitions, and model your further queries (or definitions) using them.
sirwhinesalot 4y ago
That's a good point, to be honest I'm not sure there's a particularly good answer to that beyond "it's a much better underlying model". SQL is an ugly bastardization of the relational model, whereas Datalog is a clean application of the logical model to the relational setting. One thing that's really nice about Datalog is that it puts the focus on modelling relations between data instead of "tables" which often end up with a jumble of data and then need to be normalized to work properly. It pushes you into good structures by default, which is a really good property to have. It's also much much simpler than SQL, no need to think in terms of inner joins and outer joins and whatever else, it is all relations. It also easily produces derived data from existing data, without needing any kind of procedural process, and is highly composable. It's not so much that it's solving a burning problem than SQL, it's just better than SQL... (note: the clojure syntax used in that page is much more confusing than the native Datalog syntax IMO)
- zh217 4y ago
  Another problem with Clojure-based Datalog is performance. Yet another is you are pretty much tied to the Clojure ecosystem. And I don't really like the Clojure-fused Datalog syntax either. These pains spurred me to write my own: https://docs.cozodb.org/en/latest/ https://docs.cozodb.org/en/latest/ (FOSS), and so far I'm satisified with my own work.
  - arohner 4y ago
    What is the performance problem?
yaantc 4y ago
In 2021 Google introduced Logica, a Datalog variant. In their introduction blog they contrast it with SQL, so this may answer your question: https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html https://opensource.googleblog.com/2021/04/logica-organizing-... In short, it's easier to compose (and decompose), which helps with complex queries that can be assembled from simpler, independently tested parts. No personal experience in this, I just remembered that blog contrasting Datalog and SQL.
ekidd 4y ago
> I guess the intention is to be better than SQL but then I was left with "under which circumstances?" Excellent question. Two of the most common use cases for databases are "transactional processing" (manipulating small numbers of rows in real time) and "analytical processing" (querying enormous numbers of rows, typically in a read-only fashion). SQL is generally fine for transactional workloads. But analytical queries sometimes involve multi-page queries, with lots of JOINs and CTEs. And these queries are often automatically generated. And once you start writing actual multi-page "programs" in SQL, you may decide that it's a fairly clunky and miserable programming language. What Datalog typically buys you is a way to cleanly decompose large queries into "subroutines." And it offers a simpler syntax for many kinds of complex JOINs. Unfortunately, there isn't really a standard dialect of Datalog, or even a particular dialect with mainstream traction. So choosing Datalog is a bit of a tradeoff: does it buy you enough, for your use case, that it's worth being a bit outside the mainstream? Maybe! But I'd love to see something like Logica gain more traction: https://logica.dev/ https://logica.dev/
anon291 4y ago
Sql doesn't compose at all. You can't take two sql programs and combine them with a single operator and get a new program. In datalog you can. Either by using the comma or semicolon operator.
- crabbone 4y ago
  You are thinking about the declarative part. SQL/STP, the standard that's the basis for definition of the imperative part of SQL (stored procedures) can compose in a similar way (eg. you can call a stored procedure from a stored procedure), but the standard doesn't go far enough, and so every practical database has its own extensions which deters people from using stored procedures because they'd end up with non-portable code.
  - anon291 4y ago
    stored procedures, being procedural, are the antithesis of a declarative database library. I'm talking about this: Query 1: SELECT a, b, c FROM table1 JOIN table2 ... Query 2: SELECT d, e, f FROM (SUBSELECT ... ) AS table3 JOIN table4 ... Now, join those queries together. How? Well you have to analyze the join clauses in from and then combine those together, then rewrite the SELECTION. Okay, great... not so hard. Now use Query 2 in a recursive CTE for Query 1. Oh goodness... much harder. Or what about: Query 1: SELECT a, b, c FROM table1 JOIN table2 ... Query 2: SELECT d, e, f FROM (SUBSELECT ... ) AS table3 JOIN table4 ... ORDER BY table4.column LIMIT 10 Now join 1 and 2 together... the syntax is much much much different. Some SQL libraries in Haskell, like Beam, Opaleye, Squeal, etc, demonstrate the kind of composability I mean. In particular, all three of these offer a monadic interface where arbitrary queries can be joined using well-known relational operators (relational cross joins form a monad). I'm not sure of Opaleye and Squeal, but Beam does it's best to produce a human-comprehensible query. Moreover, the first queries given above are 'easy' in the sense that the portion after the `FROM` in a regular SQL SELECT does actually compose well, since the JOIN operators are basically a direct translation of the underlying relational algebra. However, they start to fail when you want to join up aggregations and the like. While true that aggregations and ordering are not strictly relational, it is of great use to be able to compute with such things. Again, the libraries above demonstrate the composability. For example, the beam library offers this: https://haskell-beam.github.io/beam/user-guide/queries/aggregates/ https://haskell-beam.github.io/beam/user-guide/queries/aggre... Notice how we are able (in the second to last query), simply combine the query `all_ (track chinookDb)` with the complex aggregate expression. In this DSL, the entire aggregate could be floated out. In particular, notice how the queries for either the first `all_` or the second complex aggregate would look much different on their own. That's what I mean by SQL doesn't compose. Of course all these DSLs have the problem that they're trying to work with an existing language SQL that is not nice and orthogonal, and so they all have their varying deficiencies. Datalog mostly fixes these issues and provides a nice model.
felixyz 4y ago
> then I was left with "under which circumstances?" Foremost, Datalog has great compositionality, which is one of SQL:s big weaknesses (it was designed with completely different goals). Also, things like recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions.
- refset 4y ago
  > Datalog has great compositionality, which is one of SQL's big weaknesses There's a good write-up on this aspect (amongst others) here: https://www.scattered-thoughts.net/writing/against-sql/ https://www.scattered-thoughts.net/writing/against-sql/ > recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions And similarly: https://github.com/frankmcsherry/blog/blob/master/posts/2022-12-25.md https://github.com/frankmcsherry/blog/blob/master/posts/2022...
  - felixyz 4y ago
    Of all the anti-SQL screeds out there, this is my favorite as well. And Frank McSherry... well, everything he writes is gold.
dgb23 4y ago
One of the biggest advantages is re-use, extensibility and being robust to change, because you define things bottom up. Think of aggregates as "hashmaps with typed keys" rather than "structs with places in them". Aggregates are implicit groupings, not explicit slots. This wards you from risk of change and speeds up your feedback loop of making changes and experimenting with ideas. Plus you can compose new aggregate variants out of the same primitives (attributes) which is a fairly typical use-case. Another is query composition. I'm more familiar with SQL and I'm somewhat comfortable with writing nested queries. But we all know how both query builders and similar are often hard to re-use and compose. With these datalog dialects, composition, refactoring and extraction of logical units is much more straight forward, even as a beginner. Queries are "flat", and consist clauses that can be moved around and composed. Think of having more associative and commutative operations. On top of these, the mentioned technologies (in the article) are temporal and support time travel out of the box. That's something you can do on top of SQL of course but it's quite powerful to have it in-built as a primary feature. Whenever you have an application that does things like (non-exclusive), reporting, undo, revisions etc. You might want that.
crabbone 4y ago
I'm not the author, and I don't know how author would answer this question, but some things are really on the surface: 1. Modularity. In Datalog it's very easy to name a common (sub-)query and reuse it in multiple queries. You can have stored procedures / functions in SQL, but the syntax is very different implementation to implementation and users are typically afraid of using the feature (the SQL/PSM does not go far enough to define what users would normally want). Also, SQL/PSM is an imperative extension of otherwise declarative language. The two just don't work well together. 2. Structures other than primitives and tables (you get lists, vectors, hash-maps and sets, but you can also make your own). A lot of practical SQL extensions also offer some of these structures, but they aren't in the standard. 3. Operations on primitive types fall into the same category / work in the same way as operations on complex types. I.e. select / update / insert / delete in SQL only apply to tables, but strings or numbers don't work in the same way. It's more uniform in Datalog (not 100%, but still better).
conor-23 4y ago
A researchy perspective: Datalog was invented to extend relational algebra with recursion. Since it started out as an academic tool, people have been studying recursion-specific optimizations you can do for decades so it is extremely well suited to recursive use-cases e.g. iterative graph algorithms. Using Datalog for network algorithms won the thesis award in databases almost 20 years ago https://boonloo.cis.upenn.edu/papers/boon_interview.pdf https://boonloo.cis.upenn.edu/papers/boon_interview.pdf .
- tejtm 4y ago
  This is the answer I subscribe to. CTEs and recursive CTEs are SQL's answer to a limitation of plain relational algebra; no loops. CTEs are a great and most welcome addition to SQL but they are a bolt-on patch as compared with Datalog where it is a core feature.
- felixyz 4y ago
  > Datalog was invented to extend relational algebra with recursion. I'm not sure that is exactly right. Do you have a reference? (Not trying to put you on the spot, I'm just curious to learn the history!)
  - YeGoblynQueenne 4y ago
    Yeah, if the OP can give a reference I'd be very interested, too. I've searched for the "original" reference to datalog because I wanted to cite it, but I couldn't find anything like that. I have a sneaking suspicion that "function-free Prolog" is as old as ordinary Prolog, and "datalog", as an idea separate to Prolog and used as a database language, was born in the database community, but like the OP I have no reference to this.
  - refset 4y ago
    Agreed, my understanding is that Datalog has a distinct (though related) lineage that directly emerged from Prolog (i.e. logic programming, not relational algebra / database theory) - skimming the introduction of "Horn Clauses and the Fixpoint Query Hierarchy (1982)" seems to confirm this: https://dl.acm.org/doi/pdf/10.1145/588111.588137 https://dl.acm.org/doi/pdf/10.1145/588111.588137 Edit: this presentation describes things differently but it doesn't sound quite right to me "Chandra and Harel - 1982 Studied the expressive power of logic programs without function symbols on relational databases" https://www.dbai.tuwien.ac.at/datalog2.0/slides/Kolaitis.pdf https://www.dbai.tuwien.ac.at/datalog2.0/slides/Kolaitis.pdf
mmcdermott 4y ago
> I guess the intention is to be better than SQL but then I was left with "under which circumstances?". The way I see it, SQL's great strength as a query language is selection and projection whereas Datalog's great strength is inference. It takes a shift in mindset to take advantage of inference. I've started using Datalog when analyzing unfamiliar codebases. So I might set up something like: writes_to(microservice1, db1). writes_to(microservice2, microservice1). writes_to(microservice3, microservice2). depends_on(X, Y) :- writes_to(X, Z), depends_on(Z, Y). depends_on(X, Y) :- writes_to(X, Y). Allowing you to query something like: depends_on(microservice3, X) and get back db1, microservice1 and microservice2. Of course, you can ultimately do this in SQL as well, but it's far more compact in Datalog. Which brings me to the second half of the motivation - the principal advantage to the logical database model is the open world assumption. The relational model in practice tends to be fairly buttoned down - data is expected to suit a particular schema (which has its own advantages as a transactional system). This makes it easy to extend a logical database and ask more questions without changing the fact formats already specified. Of course, I can ultimately do all the things that feel natural in Datalog in SQL. I can work through queries like the one above by building out the data model and writing recursive CTEs. It's about strengths, not possibilities which I suppose brings me back to why SQL 'won'. A disproportionate amount of code is written for transactional line-of-business systems. It's really not hard to see why the validation layer and transactional focus would win in those areas.

pavlov 4y ago

As mentioned in the article, Datomic is a database that uses Datalog as its query language: https://docs.datomic.com/on-prem/query/query.html#why-datalog https://docs.datomic.com/on-prem/query/query.html#why-datalo... (Some ten years ago worked at a startup that used Datomic. It seemed to work great, although the only queries I ever needed to add to the system were simple copy-paste hacks of existing ones, so I never got to dive into Datalog.)

dmitriid 4y ago
Datascript is the open source analog for Clojure, ClojureScript and JS: https://github.com/tonsky/datascript https://github.com/tonsky/datascript
- simongray 4y ago
  There are many open source alternatives in Clojure using this query language: https://github.com/simongray/clojure-graph-resources#datalog https://github.com/simongray/clojure-graph-resources#datalog
- mpenet 4y ago
  'ish. datahike would be the closest to datomic in terms of features/implementation (support for as-of, transactor etc). Then in terms of maturity I think the choice is between xtdb and datascript, both are very solid/maintained but they are also vastly different.

tannhaeuser 4y ago

The language discussed in TFA appears to be Datomic's proprietary Clojure DSL, but has nothing to do with Datalog/Prolog.

dimitar 4y ago
It is a datalog dialect, and there a multiple open-source implementations: https://clojupedia.org/#/page/Datalog https://clojupedia.org/#/page/Datalog

thiago_fm 4y ago

The problem with Datalog, and Clojure in general are the licenses. Terrible licenses. Everything is about Rich Hickey. Apache 1.0. Now that Nubank basically owns it and there's very little progress or activity as of late, I don't see why one would chose to use Clojure, Datalog etc. Also, a lot of functional programming concepts has been since added to big programming languages like Javascript and hell, even Java has lambdas now. I'm guessing that also hardcore FP people have moved on to Haskell. The ones that like LISP to Racket... and only people tied to the JVM in legacy projects are with Clojure.

maweki 4y ago
What are the licensing issues with First-order Horn clause logic? Datalog is not a licensed product or software, the same way Answer Set Programming isn't.
casion 4y ago
> very little progress or activity as of late There's been equally frequent releases and updates, I'm not sure how you came to this conclusion? The best I can think of is that you're confusing core language updates with other tech. Clojure is structured differently than other languages, and core language/library updates are (and always have been) relatively rare while the surrounding ecosystem/tooling provided by the Clojure team is active.
armincerf 4y ago
XTDB is MIT and has clients for Java or HTTP if you don't want to write Clojure. Not every Datalog implementation is 'about Rich Hickey'
cmrdporcupine 4y ago
Datalog != Datomic. Datomic ∈ Datalog
- refset 4y ago
  It's true that this SPARQL-inspired view of Datalog as a triplestore query language is quite a narrow interpretation compared to something closer to the academic Prolog roots like https://souffle-lang.github.io/ https://souffle-lang.github.io/ - what do you feel are the most important differences?

wslh 4y ago

In the last few months the mention of Datalog has increased, I wondered how it differed from graph databases and found a clear answer in SO [1]. I am not an incumbent but found graph databases and clause approaches interesting. [1] https://stackoverflow.com/questions/29192927/a-graph-db-vs-a-prolog-or-minikanren https://stackoverflow.com/questions/29192927/a-graph-db-vs-a... (2015)

noduerme 4y ago
That's a really neat example of something I'm not familiar with. Going up a tree from child to parent is often the heaviest part of dealing with regular datasets, and usually requires a mix of queries and application logic. The idea of flattening the data along some pattern like that is of course always possible in a relational db, but it's not usually efficient, especially not for heavy writing. Lateral joins and window partitions can help. But this seems like an interesting approach to removing the app code completely.
refset 4y ago
XTDB, which is mentioned in the post, is subtly different from the other Clojure-based Datalog systems in this respect, because its Datalog engine executes in terms of multi-way joins using a "Worst-Case Optimal Join" implementation that is ideal for graph processing (vs. a tree of binary hash joins). Therefore, based on statistics and query planning heuristics, it will often perform graph pattern matching before resolving the logic/horn clauses. (source: I work on the XTDB team)
- eternalban 4y ago
  Interesting architecture: https://raw.githubusercontent.com/xtdb/xtdb/master/docs/concepts/modules/ROOT/images/xtdb-node-1.svg https://raw.githubusercontent.com/xtdb/xtdb/master/docs/conc... Btw, is that 'RocksDB or ?' for the local store current or other storage engines can get plugged in? p.s. this is datomic's architecture for comparison. https://docs.datomic.com/on-prem/images/clientarch_orig.svg https://docs.datomic.com/on-prem/images/clientarch_orig.svg
  - 4y ago

dataengineer56 4y ago

This is a cool concept but I'm not sure I'd fancy upskilling a team of SQL analysts to use this.

cmrdporcupine 4y ago
Many SQL analysts are wasting their skills performing mental and syntactical gymnastics to get around the limitations of SQL in order to grasp at the actual conceptual elegance that lays underneath it. Most people who are writing SQL for a living already understand at least some part of what makes the relational model powerful. But SQL is relatively poor tool for accessing it. I personally don't find the Sexpr-based syntax of Datomic's variant of Datalog all that useful here, and yeah, maybe someone working in SQL for a living would struggle at first with that syntax. But it's not intrinsic to the model itself. Have a waltz through my employer's documentation and see what you think: https://docs.relational.ai/rel/primer/overview https://docs.relational.ai/rel/primer/overview I think it's quite understandable (if a bit terse) and many people doing SQL for a living would appreciate the ability to better compose and structure things in this way, not to mention the ability to handle transitive / recursive relationships in a less awkward way.

eunos 4y ago

Huh cool, I didnt realize it come from that Michelin.

eddieroger 4y ago
I remember being surprised once that Michelin, like the star, was the same as Michelin, like the tire. It's really cool to see a company move beyond their core competency more than once in a meaningful way, even if the first time was to sell more tires. They have a really interesting blog that this is just a single article from.
- pbronez 4y ago
  I believe they started the star ratings as a way to encourage people to drive more. They saw it as part of “Travel”.
  - julienchastang 4y ago
    Yes, absolutely, as are the eponymous "Guide Michelin" (Guide vert) that conveniently fit in the glove compartment of your automobile. More people drive, more they consume tires.

k4st 4y ago

I created a datalog engine a few years back called Dr. Lojekyll: https://www.petergoodman.me/docs/dr-lojekyll.pdf https://www.petergoodman.me/docs/dr-lojekyll.pdf It was pretty cool; you could stream in new facts to it over time and it would incrementally and differentially update itself. The key idea was that I wanted the introduction of ground facts to be messages that the database reads (e.g. off of a message bus), and I wanted the database to be able to publish its conclusions onto the same or other message buses. I also wanted to be able to delete ground facts, which meant it could publish withdrawals of the prior-published conclusions. A lot of it was inspired by Frank McSherry's work, although I didn't use timely or differential dataflow. In retrospect I probably should have! This particular system isn't used anymore because we made a classic monotonicity mistake by making it the brain of a distributed system, and then having it publish and receive messages with a bunch of microservices. The internal consistency model of the datalog engine didn't extend out to the microservices, and the possibility of feedback loops in the system meant that the whole thing could lie to itself and diverge uncontrollably! Despite this particular application of the engine being a failure, the engine itself worked quite well and I hope to one day return to datalog. I think what a lot of people miss with datalog, and what becomes apparent as you use it more, is just how unpredictable many engines can be with the execution behavior of rules. This is the same problem that you have with a database, where the query planner makes a bad choice or where you lack an index, and so performance is bad. But with datalog, the composition of rules that comes so naturally also tends to compound this issue, resulting in time spent trying to chase down weird performance things and doing spooky re-ordering of your clause bodies to try to appease whatever choices the engine makes.

samuell 4y ago

A bit related, just stumbled upon Flix, a functional JVM language with Datalog contraints and (somewhat?) Go-like concurrency: https://flix.dev https://flix.dev HN Thread from 8 months ago: https://news.ycombinator.com/item?id=31448889 https://news.ycombinator.com/item?id=31448889

refset 4y ago
Flix definitely looks interesting! For comparison, I ported the "Datalog Enriched with Lattice Semantics" example from that homepage to XTDB's (Clojure) Datalog after I saw it posted on HN originally: https://gist.github.com/refset/21b3fc1dec9a6928943073809e13356d https://gist.github.com/refset/21b3fc1dec9a6928943073809e133...

muattiyah 4y ago

ICYMI, there's an excellent interactive introduction to `datalog` that's referenced in the article's references.[0] Last time I used `datalog` was years ago, I was developing an internal interactive tool that was used to compare different approaches to solving a certain problem at my employer. I used `datascript`[1] by way of clojurescript to store all experiment data and then interrogated the `datascript` DB via `datalog`. This is something I always remember fondly. [0] https://www.learndatalogtoday.org/ https://www.learndatalogtoday.org/ [1] https://github.com/tonsky/datascript https://github.com/tonsky/datascript

anon291 4y ago

Whatever language this is... This is not datalog. This looks like a particular implementation of datalog in closure. Actual datalog looks like prolog.

cmrdporcupine 4y ago
In reality, people are using "datalog" for a genre of datastore concepts based around horne clauses, or, basically relations + implicit joins. Datalog as a subset or dialect of prolog is only one variant of this. And Datomic has made an sexpr-syntaxed variant built around binary relations popular. To the point where some people in this thread can't seem to tell the two apart. I am more interested in the general category of relational data model + logic programming than I am in any purity about Datalog in particular. In particular I'm very excited by "data/knowledge + behaviour sitting in a tree, k-i-s-s-i-n-g"
- bogomipz 4y ago
  Thanks. Is Datomic also a dialect of Prolog then?
  - cmrdporcupine 4y ago
    No, and I wouldn't call Datalog a dialect of Prolog either necessarily. It's more like, classic Datalog syntax looks Prolog-ish (similar syntax, and is kind of a subset of Prolog). And you can do Datalog-type stuff in Prolog. But from what I've read of Datomic it's got a totally different syntax, but similar semantics to Datalog .. but not Prolog... Basically, Prolog can do more than Datalog. Datalog is like, some subset of concepts of Prolog, but specialized for relational data queries, so it can achieve some optimizations and focus on data retrieval only. Is that confusing enough?
    - dragonwriter 4y ago
      > No, and I wouldn’t call Datalog a dialect of Prolog either necessarily. It’s more like, classic Datalog syntax looks Prolog-ish (similar syntax, and is kind of a subset of Prolog). Datalog originated specifically as a restricted subset of Prolog, which is why the “classic” syntax looks prolog-ish. > Basically, Prolog can do more than Datalog. Datalog is like, some subset of concepts of Prolog, but specialized for relational data queries, so it can achieve some optimizations and focus on data retrieval only. Datalog is a purely declarative, non-Turing complete subset of Prolog, removing the imperative features and assuring termination. This gives up lots of power, obviously, but it also, well, provides a termination guarantee. Removing imperative features (cut) from Prolog also means that Datalog implementations can use different (or multiple, switchable) strategies, while differing in what kinds of data sets and queries they perform well on, not correctness.

i_am_toaster 4y ago

Maybe it’s just me but I find the SQL much easier to read in all the examples given.

grose 4y ago

Datalog is great for representing authorization rules. Check out Biscuits, which are auth tokens with Datalog embedded in them. This article is what made it 'click' for me: https://www.clever-cloud.com/blog/engineering/2021/04/15/biscuit-tutorial/ https://www.clever-cloud.com/blog/engineering/2021/04/15/bis... I actually thought that Datalog was so cool that I went to learn Prolog and it completely changed the way I think about programming. Highly recommend trying out logic programming if you haven't before.

burakemir 4y ago
Agree but also want to point out that people usually have a narrow view on "logic programming". Datalog can also be understood with out the top-down evaluation / resolution that is typically associated with prolog, which is why it is known to database researchers and in finite model theory. Prolog is great, but bottom up techniques to evaluate datalog are awesome, too, and would arguably also qualify as logic programming. It is rare to see this acknowledged.
- YeGoblynQueenne 4y ago
  Acknowledged, by whom? As far as I know people in the logic programming community have no trouble recognising datalog as a logic programming language, be it evaluated bottom-up or not (you can still evaluate a datalog program top-down, by resolution, as if it were a Prolog program without "compound" terms) (a.k.a. functions). Indeed, I get the feeling that the primacy of Prolog as the logic programming language has waned. If you look at back issues of ICLP* proceedings you'll find plenty of work that is nothing to do with resolution- namely, Answer Set Programming, which is wildly popular. I tend to think it's mainly the database community that kind of ignores the logic-programming nature of datalog. Oh and btw, when I talk about datalog, I mean definite clauses without functions, not the aberrant syntax in the article above. I'll never understand why people do that. ________________ * ICLP is the International Conference on Logic Programming.

z5h 4y ago

I’ve been using Prolog a bunch recently, and also embedded and extended MicroKanren in a project. Something I came to appreciate was that Prolog’s depth-first search, and Kanren’s lazy stream approach are good with memory even when generating/searching through infinite solutions. It is my understanding that Datalog, on the other hand, will iteratively expand a set of data. Isn’t this a problem?

YeGoblynQueenne 4y ago
"Iteratively expand a set of data"? I'm not sure what you mean here. I think you are probably talking about the "bottom up" evaluation strategy of datalog, right? That's where datalog is evaluated by a so-called TP-operator, which derives the set of all logical consequences a datalog program by calculating its least fixed point. That's the same as the Least Herbrand Model (LHM) of the program, or, in other words, the set of atoms entailed by the program (atoms in the logical sense, of atomic formulae, not in the Prolog sense of constants). That's the same thing that Prolog does, calculate the LHM of a logic program, but the difference is that datalog programs have finite LHMs, because they don't have functions that can be self-instantiated for ever ( f(x), f(f(x)), f(f(f(x))), ... ) and the bottom-up evaluation, that goes from the "body" to the "head" of a clause, avoids infinite left-recursions. Prolog, evaluated top-down (clauses are "picked apart" head-first) can get stuck in infinite left-recursions, so datalog's finiteness, and its decidability under TP, is a big gain in efficiency, as anyone who has had to kill a Prolog console session because of an infinite loop will know. Also, it is not widely recognised but I am the author of the dumbest and most inefficient TP Operator implementation in existence. Obviously I hang my head in shame and will not link to my code. I understand however that there are optimisations that one can perform that make bottom-up execution efficient, and even quite fast. Unfortunately, I don't know what they are :P Note that Prolog can also be evaluated without fear of left-recursions, by SLG-Resolution (a.k.a. "tabling", a.k.a. memoization) but there is still the danger of infinite right recursions. Prolog is semi-decidable, because it is Turing-complete. Datalog is decidable, sacrificing completeness for, well, efficiency. So, in short, it's not a problem if you consider the alternative, but of course there are trade-offs, always. It's like growing old, vs. dying young. (I hope all this is not completely irrelevant to your question).