8 ms·

How I have waited for this: A simple, accessible library for graph-like data with datalog (also in a statically compiled language, yay). Have even pondered usin

How I have waited for this: A simple, accessible library for graph-like data with datalog (also in a statically compiled language, yay). Have even pondered using SWI-prolog for this kind of stuff, but it seems so much nicer to be able to use it embedded in more "normal" types of languages. Looking forward to play with this! The main thing I will be wondering now is how it will scale to really large datasets. Any input on that?

zh217 4y ago
Thanks for your interest in this! It currently uses RocksDB as the storage engine. If your server has enough resources, I believe it can store TBs of data with no problem. Running queries on datasets this big is a complicated story. Point lookups should be nearly instant, whereas running complicated graph algorithms on the whole dataset is (currently) out of the question, since all the rows a query touches must reside in memory. Also, the algorithmic complexity of some of the graph algorithms is too high for big data and there's nothing we can do about it. We aim to provide a smooth way for big data to be distilled layer by layer, but we are not there yet.
- samuell 4y ago
  Many thanks for the detailed answer!
- bryanrasmussen 4y ago
  when you say currently it implies it will change? Does that mean all rows will not be in memory? what if you had not so many nodes but each node had a lot of data would that improve it? Probably not but just normally I think of the number of nodes in your graph as the problem.
  - zh217 4y ago
    Yes. For example, in Postgres you can sort tables arbitrarily large, not constrained by main memory. Postgres uses external merge sort when the tables are really large. There are other situations where the working data are disk-based when they are too large in Postgres. We will eventually be able to do that in Cozo as well, but no timetable is available yet. For your second question, say you have a relation with lots of fields, one of them particularly large. As long as you don't use that field in your query, it will not impact memory usage. The query may be slower though since the RocksDB storage engine needs to read more pages from disk, but the fields that are loaded by RocksDB but not needed will be promptly evicted from memory.
samuell 4y ago
For folks looking for documentation or getting started-examples, see: - The tutorial: https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutorial/tutorial.ipynb https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutor... - The language documentation: https://cozodb.github.io/current/manual/ https://cozodb.github.io/current/manual/ - The pycozo library README for some examples on how to run this from inline python: https://github.com/cozodb/pycozo#readme https://github.com/cozodb/pycozo#readme

zh217 4y ago

Thanks for your interest in this! It currently uses RocksDB as the storage engine. If your server has enough resources, I believe it can store TBs of data with no problem. Running queries on datasets this big is a complicated story. Point lookups should be nearly instant, whereas running complicated graph algorithms on the whole dataset is (currently) out of the question, since all the rows a query touches must reside in memory. Also, the algorithmic complexity of some of the graph algorithms is too high for big data and there's nothing we can do about it. We aim to provide a smooth way for big data to be distilled layer by layer, but we are not there yet.

samuell 4y ago
Many thanks for the detailed answer!
bryanrasmussen 4y ago
when you say currently it implies it will change? Does that mean all rows will not be in memory? what if you had not so many nodes but each node had a lot of data would that improve it? Probably not but just normally I think of the number of nodes in your graph as the problem.
- zh217 4y ago
  Yes. For example, in Postgres you can sort tables arbitrarily large, not constrained by main memory. Postgres uses external merge sort when the tables are really large. There are other situations where the working data are disk-based when they are too large in Postgres. We will eventually be able to do that in Cozo as well, but no timetable is available yet. For your second question, say you have a relation with lots of fields, one of them particularly large. As long as you don't use that field in your query, it will not impact memory usage. The query may be slower though since the RocksDB storage engine needs to read more pages from disk, but the fields that are loaded by RocksDB but not needed will be promptly evicted from memory.

samuell 4y ago

For folks looking for documentation or getting started-examples, see: - The tutorial: https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutorial/tutorial.ipynb https://nbviewer.org/github/cozodb/cozo-docs/blob/main/tutor... - The language documentation: https://cozodb.github.io/current/manual/ https://cozodb.github.io/current/manual/ - The pycozo library README for some examples on how to run this from inline python: https://github.com/cozodb/pycozo#readme https://github.com/cozodb/pycozo#readme