4 ms·

I just spent ten minutes looking through the repo. I like how the query language is similar enough to Datomic’s to have an easy learning curve. I wish I could p

I just spent ten minutes looking through the repo. I like how the query language is similar enough to Datomic’s to have an easy learning curve. I wish I could push a magic button and have this in Common Lisp (where I usually just use SQLite as an easy to use data store).

quoll 4y ago
The main problem is that I don't have a lot of time on it right now, though I'm trying to do more. I don't actually know Common Lisp, though it seems to work nicely when compiled to binary via Graal, so maybe there's some possibilities of integration using that approach?
harryvederci 4y ago
An idea that pops up more and more often in my head is to create a SQLite wrapper library that basically treats SQLite like Datalog. I know way too little about Datalog, but it goes something like this: 1. create helper functions to create/alter/drop tables. 2. create helper functions for insert/select/update/delete queries, etc. Probably with the same syntax as in Datomic / Asami / etc. 3. each table its name would be a Datalog "attribute", and would have 2 columns: the "id" and the "value". "id" would be indexed, "value" would be a value or a foreign key to another table. I guess writes would be slow if everything you insert into the DB is indexed, not sure what the other downsides would be.
- packetlost 4y ago
  It's not like datomic claims to be lightening fast, it's not. The real problem is if you want to support the historical queries "as_of()", which is very difficult to do efficiently with only B-tree indexes. So you're either left maintaining your own indices as tables or externalizing it to something else. I think the latter would actually be a pretty ok option if your dataset fits entirely in memory (though... for historical queries it will grow much faster than a normal database). Regardless, I have a partial attempted implementation of this on top of SQLite using Python: https://git.sr.ht/~chiefnoah/quark https://git.sr.ht/~chiefnoah/quark No query engine, I didn't get that far.
  - quoll 4y ago
    Not sure what you mean here? as_of() is easy to do efficiently, so long as you have tree indexes (it doesn't really matter what type of tree, though fractal trees would not be good). But you need direct access to the tree, rather than having an index that's built using a tree, since then you can't build the phases.
    - packetlost 4y ago
      Right, if you have direct access to the tree it's possible and not really inefficient at all (a modified B+Tree would be my first pick, but I digress). Part of my problem was I was trying to offload as much of the work to SQLite as possible. I think if I were to go back and try again, I would make it a bit further :)
  - harryvederci 4y ago
    I'm talking about Datalog, I mentioned Datomic purely in the context of syntax. And to sum up what I know about Datalog, it's basically just this article (not reference material, but it got me interested in the concept), plus some light wikipedia-ing: https://www.instantdb.com/essays/datalogjs https://www.instantdb.com/essays/datalogjs Edit: I think you added the part with your git repo after I wrote this. Looks cool, could you add a license file to it?
    - packetlost 4y ago
      Got it. Yeah, datalog is cool. I really like it (if that wasn't obvious from my attempt at implementing it). While not built on SQLite, this project has caught my attention recently: https://github.com/cozodb/cozo https://github.com/cozodb/cozo It's built on RocksDB and has slightly different query syntax (supposedly to be more similar to Python's, as a primary target usecase is within Jupyter notebooks). Check it out!
      - zh217 4y ago
        More precisely, Cozo currently has a RocksDB backend. The development version also has SQLite backend (for easier compilation on mobile platforms), in-memory backend, etc. It now can run as a web assembly module as well (see https://cozodb.github.io/wasm-demo/ https://cozodb.github.io/wasm-demo/) With respect to the SQLite backend, Cozo uses SQLite purely as a KV store with range scan capabilities, with both keys and values stored as bytes. This actually results in faster queries than offloading some stuff to the SQLite engine for the (very few) cases we tested. Disclaimer: I'm the author of Cozo.
- michaelsbradley 4y ago
  Did you see the Show HN post announcing cozo 10 days ago? https://news.ycombinator.com/item?id=33518320 https://news.ycombinator.com/item?id=33518320 https://github.com/cozodb/cozo https://github.com/cozodb/cozo Note: there was a fair amount of discussion about the license; the author eventually switched it to MPL2.
- dms92n1bx 4y ago
  There is (now unmaintained) project called Mentat [0] from Mozilla. [0] https://github.com/mozilla/mentat https://github.com/mozilla/mentat
fiddlerwoaroof 4y ago
Do you happen to know of any good resources for implementing datalog? I’ve thought about writing something similar in Common Lisp several times, but it’s hard to find a good introduction here.
sitkack 4y ago
My first thought was a Truffle CommonLisp so you could integrate via the GraalVM. https://github.com/charig/TCLisp https://github.com/charig/TCLisp https://github.com/armedbear/abcl/issues/62 https://github.com/armedbear/abcl/issues/62

quoll 4y ago

The main problem is that I don't have a lot of time on it right now, though I'm trying to do more. I don't actually know Common Lisp, though it seems to work nicely when compiled to binary via Graal, so maybe there's some possibilities of integration using that approach?

harryvederci 4y ago

An idea that pops up more and more often in my head is to create a SQLite wrapper library that basically treats SQLite like Datalog. I know way too little about Datalog, but it goes something like this: 1. create helper functions to create/alter/drop tables. 2. create helper functions for insert/select/update/delete queries, etc. Probably with the same syntax as in Datomic / Asami / etc. 3. each table its name would be a Datalog "attribute", and would have 2 columns: the "id" and the "value". "id" would be indexed, "value" would be a value or a foreign key to another table. I guess writes would be slow if everything you insert into the DB is indexed, not sure what the other downsides would be.

packetlost 4y ago
It's not like datomic claims to be lightening fast, it's not. The real problem is if you want to support the historical queries "as_of()", which is very difficult to do efficiently with only B-tree indexes. So you're either left maintaining your own indices as tables or externalizing it to something else. I think the latter would actually be a pretty ok option if your dataset fits entirely in memory (though... for historical queries it will grow much faster than a normal database). Regardless, I have a partial attempted implementation of this on top of SQLite using Python: https://git.sr.ht/~chiefnoah/quark https://git.sr.ht/~chiefnoah/quark No query engine, I didn't get that far.
- quoll 4y ago
  Not sure what you mean here? as_of() is easy to do efficiently, so long as you have tree indexes (it doesn't really matter what type of tree, though fractal trees would not be good). But you need direct access to the tree, rather than having an index that's built using a tree, since then you can't build the phases.
  - packetlost 4y ago
    Right, if you have direct access to the tree it's possible and not really inefficient at all (a modified B+Tree would be my first pick, but I digress). Part of my problem was I was trying to offload as much of the work to SQLite as possible. I think if I were to go back and try again, I would make it a bit further :)
- harryvederci 4y ago
  I'm talking about Datalog, I mentioned Datomic purely in the context of syntax. And to sum up what I know about Datalog, it's basically just this article (not reference material, but it got me interested in the concept), plus some light wikipedia-ing: https://www.instantdb.com/essays/datalogjs https://www.instantdb.com/essays/datalogjs Edit: I think you added the part with your git repo after I wrote this. Looks cool, could you add a license file to it?
  - packetlost 4y ago
    Got it. Yeah, datalog is cool. I really like it (if that wasn't obvious from my attempt at implementing it). While not built on SQLite, this project has caught my attention recently: https://github.com/cozodb/cozo https://github.com/cozodb/cozo It's built on RocksDB and has slightly different query syntax (supposedly to be more similar to Python's, as a primary target usecase is within Jupyter notebooks). Check it out!
    - zh217 4y ago
      More precisely, Cozo currently has a RocksDB backend. The development version also has SQLite backend (for easier compilation on mobile platforms), in-memory backend, etc. It now can run as a web assembly module as well (see https://cozodb.github.io/wasm-demo/ https://cozodb.github.io/wasm-demo/) With respect to the SQLite backend, Cozo uses SQLite purely as a KV store with range scan capabilities, with both keys and values stored as bytes. This actually results in faster queries than offloading some stuff to the SQLite engine for the (very few) cases we tested. Disclaimer: I'm the author of Cozo.
michaelsbradley 4y ago
Did you see the Show HN post announcing cozo 10 days ago? https://news.ycombinator.com/item?id=33518320 https://news.ycombinator.com/item?id=33518320 https://github.com/cozodb/cozo https://github.com/cozodb/cozo Note: there was a fair amount of discussion about the license; the author eventually switched it to MPL2.
dms92n1bx 4y ago
There is (now unmaintained) project called Mentat [0] from Mozilla. [0] https://github.com/mozilla/mentat https://github.com/mozilla/mentat

fiddlerwoaroof 4y ago

Do you happen to know of any good resources for implementing datalog? I’ve thought about writing something similar in Common Lisp several times, but it’s hard to find a good introduction here.

sitkack 4y ago

My first thought was a Truffle CommonLisp so you could integrate via the GraalVM. https://github.com/charig/TCLisp https://github.com/charig/TCLisp https://github.com/armedbear/abcl/issues/62 https://github.com/armedbear/abcl/issues/62