4 ms·

Many programming languages are wonderful for working with data, are you sure you need to go full Datalog? Restricting yourself to a simple subset of Haskell or

Many programming languages are wonderful for working with data, are you sure you need to go full Datalog? Restricting yourself to a simple subset of Haskell or using something like Clojure itself (for which you can find Datalog libraries btw) to build your query engine seems like it would offer much of the same expressive declarativity of Datalog.

samuell 10y ago
Thanks! Datalog library for clojure is something I should definitely look up. Will think hard about the suggestion of using some functional language instead also ... you for sure got me thinking! :) Feel free to share any pointers regarding this! In the meanwhile, sharing my thoughts so far: What I have so far thought was pretty "unique" about datalog is that it support "named queries", which can be used as a kind of "duck typing" classification, to build up a model of your domain of interest that is always computed on-demand, leaving the raw data unchanged in it's default ("normal"?) form. Because of the "named queries", you can combine lower-level queries in higher-level ones, to create a hierarchy of "dynamically generated concepts", which I haven't seen any other technology be able to do. (While named queries are conceptually similar to views in SQL, they are so much more natural to work with.) I know this by experience from my dabblings in prolog [1], but haven't found any large-scale prolog data system supporting distributed computing (I understood it has some features making it unpractical for distributed computing and more, in a way that datalog doesn't). [1] http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A398839&dswid=2235 http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A3988... (PDF: http://www.diva-portal.org/smash/get/diva2:398839/FULLTEXT01.pdf http://www.diva-portal.org/smash/get/diva2:398839/FULLTEXT01... )
- samuell 10y ago
  Well, thinking about it, I guess one could achieve aspects of this by using lazy iterators in a functional programming language too, but it still seems to lack the declarative nature that you get with the unification / pattern matching of datalog/prolog ... I might be wrong though.
  - notduncansmith 10y ago
    I know exactly what you mean. Read this section from the Elixir docs: http://elixir-lang.org/getting-started/pattern-matching.html http://elixir-lang.org/getting-started/pattern-matching.html Also this page from the Haskell tutorial: https://www.haskell.org/tutorial/patterns.html https://www.haskell.org/tutorial/patterns.html Fun fact: early versions of Erlang (which Elixir is built upon) were implemented in Prolog, and IIRC what would become Erlang later was originally intended to be an extension of Prolog. There are no new ideas.
- jamii 10y ago
  You can build your own foward-chaining datalog compiler pretty easily - http://scattered-thoughts.net/blog/2016/10/11/a-practical-relational-query-compiler-in-500-lines/ http://scattered-thoughts.net/blog/2016/10/11/a-practical-re.... I did it in-memory only, but that same design would work with any external storage that supports range queries efficiently.
  - samuell 10y ago
    Thanks tons, I've been searching hard for content like this!
- notduncansmith 10y ago
  > Because of the "named queries", you can combine lower-level queries in higher-level ones, to create a hierarchy of "dynamically generated concepts", which I haven't seen any other technology be able to do. This is the purpose of functions in most programming languages. A function is kind of like an HTML template, but for code. So rather than have `<h1 title="{{bar}}">{{foo}}</h1>` you have `function(foo, bar) { return h1({title: bar, text: foo}); }`. DOM tree, AST ("abstract syntax tree"), relational data, same thing: a graph, and most programming languages can represent raw graph nodes/edges easily as lists of strings. You can write a simple tuple-matching engine in Javascript: https://gist.github.com/notduncansmith/7858d27a2b875ff10b84 https://gist.github.com/notduncansmith/7858d27a2b875ff10b84 Obviously that's not a production-ready piece of software, but it highlights an important lack of distinction between "database" and "program" that wasn't readily apparent to me as a developer for a while. HTML, source code, and user input are all the same thing (just data, really). One program's source code is another's input data, all the way down. If you want to get existential, you can extend this even down to matter itself as simply a representation of information: imagine a physical machine in a factory that winds jack-in-the-box toys on their way out the door. Same thing as a higher-order function. function wind(toy) { return toy.lever.rotateByDegrees(360); } So basically, code is data is matter, your source code is just electricity in a machine and whether you look at it as "code" or "data" is simply a matter of perspective. This is what Lisp folks mean by "code is data". Moving from the abstract to your concrete case: > build up a model of your domain of interest that is always computed on-demand, leaving the raw data unchanged in it's default ("normal"?) form. [...] (While named queries are conceptually similar to views in SQL, they are so much more natural to work with.) SQL views don't compose at the same level that Datalog expressions do, but that's because they aren't typically expressed as raw graph data (i.e. lists of strings where each list item is a node in the graph). If you could just name arbitrary chunks of SQL AST and mash them together, it would be just as composable (though perhaps more cumbersome due to boilerplate). One final thing before this already-too-long comment comes to a close: you might give CouchDB a look. It's a document datastore that offers map/reduce views written in Javascript (use as much or as little composition/abstraction as you want). The distributed features are pretty cool (bidirectional sync, versioned documents) and writing apps with it is awesome, especially if you can leverage supporting projects like PouchDB or Couchbase Lite. Not sure if it fits your use-case but it sounds like you might find it personally interesting. Hopefully this has been helpful!

samuell 10y ago

Thanks! Datalog library for clojure is something I should definitely look up. Will think hard about the suggestion of using some functional language instead also ... you for sure got me thinking! :) Feel free to share any pointers regarding this! In the meanwhile, sharing my thoughts so far: What I have so far thought was pretty "unique" about datalog is that it support "named queries", which can be used as a kind of "duck typing" classification, to build up a model of your domain of interest that is always computed on-demand, leaving the raw data unchanged in it's default ("normal"?) form. Because of the "named queries", you can combine lower-level queries in higher-level ones, to create a hierarchy of "dynamically generated concepts", which I haven't seen any other technology be able to do. (While named queries are conceptually similar to views in SQL, they are so much more natural to work with.) I know this by experience from my dabblings in prolog [1], but haven't found any large-scale prolog data system supporting distributed computing (I understood it has some features making it unpractical for distributed computing and more, in a way that datalog doesn't). [1] http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A398839&dswid=2235 http://www.diva-portal.org/smash/record.jsf?pid=diva2%3A3988... (PDF: http://www.diva-portal.org/smash/get/diva2:398839/FULLTEXT01.pdf http://www.diva-portal.org/smash/get/diva2:398839/FULLTEXT01... )

samuell 10y ago
Well, thinking about it, I guess one could achieve aspects of this by using lazy iterators in a functional programming language too, but it still seems to lack the declarative nature that you get with the unification / pattern matching of datalog/prolog ... I might be wrong though.
- notduncansmith 10y ago
  I know exactly what you mean. Read this section from the Elixir docs: http://elixir-lang.org/getting-started/pattern-matching.html http://elixir-lang.org/getting-started/pattern-matching.html Also this page from the Haskell tutorial: https://www.haskell.org/tutorial/patterns.html https://www.haskell.org/tutorial/patterns.html Fun fact: early versions of Erlang (which Elixir is built upon) were implemented in Prolog, and IIRC what would become Erlang later was originally intended to be an extension of Prolog. There are no new ideas.
jamii 10y ago
You can build your own foward-chaining datalog compiler pretty easily - http://scattered-thoughts.net/blog/2016/10/11/a-practical-relational-query-compiler-in-500-lines/ http://scattered-thoughts.net/blog/2016/10/11/a-practical-re.... I did it in-memory only, but that same design would work with any external storage that supports range queries efficiently.
- samuell 10y ago
  Thanks tons, I've been searching hard for content like this!
notduncansmith 10y ago
> Because of the "named queries", you can combine lower-level queries in higher-level ones, to create a hierarchy of "dynamically generated concepts", which I haven't seen any other technology be able to do. This is the purpose of functions in most programming languages. A function is kind of like an HTML template, but for code. So rather than have `<h1 title="{{bar}}">{{foo}}</h1>` you have `function(foo, bar) { return h1({title: bar, text: foo}); }`. DOM tree, AST ("abstract syntax tree"), relational data, same thing: a graph, and most programming languages can represent raw graph nodes/edges easily as lists of strings. You can write a simple tuple-matching engine in Javascript: https://gist.github.com/notduncansmith/7858d27a2b875ff10b84 https://gist.github.com/notduncansmith/7858d27a2b875ff10b84 Obviously that's not a production-ready piece of software, but it highlights an important lack of distinction between "database" and "program" that wasn't readily apparent to me as a developer for a while. HTML, source code, and user input are all the same thing (just data, really). One program's source code is another's input data, all the way down. If you want to get existential, you can extend this even down to matter itself as simply a representation of information: imagine a physical machine in a factory that winds jack-in-the-box toys on their way out the door. Same thing as a higher-order function. function wind(toy) { return toy.lever.rotateByDegrees(360); } So basically, code is data is matter, your source code is just electricity in a machine and whether you look at it as "code" or "data" is simply a matter of perspective. This is what Lisp folks mean by "code is data". Moving from the abstract to your concrete case: > build up a model of your domain of interest that is always computed on-demand, leaving the raw data unchanged in it's default ("normal"?) form. [...] (While named queries are conceptually similar to views in SQL, they are so much more natural to work with.) SQL views don't compose at the same level that Datalog expressions do, but that's because they aren't typically expressed as raw graph data (i.e. lists of strings where each list item is a node in the graph). If you could just name arbitrary chunks of SQL AST and mash them together, it would be just as composable (though perhaps more cumbersome due to boilerplate). One final thing before this already-too-long comment comes to a close: you might give CouchDB a look. It's a document datastore that offers map/reduce views written in Javascript (use as much or as little composition/abstraction as you want). The distributed features are pretty cool (bidirectional sync, versioned documents) and writing apps with it is awesome, especially if you can leverage supporting projects like PouchDB or Couchbase Lite. Not sure if it fits your use-case but it sounds like you might find it personally interesting. Hopefully this has been helpful!