3 ms·
Jena lets you make little in-memory triple stores that you can use the way people use the list-map-scalar trinity. I've been working on this publication about
by PaulHoule 2y ago
Jena lets you make little in-memory triple stores that you can use the way people use the list-map-scalar trinity. I've been working on this publication about that (RDF for difficult cases and when ordering counts) for years and it just got published last week
https://www.iso.org/standard/76310.html https://www.iso.org/standard/76310.html
I'll call out my collabortor Liju Fan for being the only person I've met who knew how to do anything interesting with OWL. (Well, I can do interesting things now but I owe it all to her.)
(For the research for that paper I used rdflib under PyPi because CPython was not fast enough.)
When I needed big persistent triple stores (that you use the way you might use postgres) I used to use
https://en.wikipedia.org/wiki/Virtuoso_Universal_Server https://en.wikipedia.org/wiki/Virtuoso_Universal_Server
and had pretty good luck if I loaded a billion triples if I used plenty of 'stabilizers' (create a new AWS instance with ample RAM, use scripts to load a billion triples starting from an empty database, shut it down, make an AMI, start a new instance with the AMI, expect it to warm up for 20 minutes or so before query performance is good)
I don't regularly build systems on SPARQL today because of problems with updating. In particular, SQL has an idea of a "record" which is a row in a table, document oriented databases have an idea of a "record" which is a bit more flexible. Updating a SPARQL database is a little bit dangerous because there is no intrinsic idea of what a record is; i mean, you can define one by starting at a particular URI and traversing to the right across blank nodes and saying it is a 'record' and it works OK. But it's a discipline that I impose on it with my libraries, it ought to be baked into standards, baked into the databases, wrapped up in transactions, etc. For anything OLTP-ish I am still using SQL or document-oriented databases, but I hate the lack of namespaces and similar affordances that make SPARQL scalable in terms of "smash together a bunch of data from different sources" in document-oriented databases wheras SPARQL is missing the affordances you have in document-oriented databases for handling ordered collections. We badly need a SPARQL 2 which makes the kind of work that I talk about in that technical report easy.
- svilen_dobrev 2y agodatomic (and partially xtdb /former crux) are OLTPish, and use only such "tuples" , essentially it's up to the user to define what constitutes an entity if at all ("row", "object", "document", whatever) - maybe some entity-id and everything linked to it, but maybe other less-identity-related stuff. Which might feel freeing to extent, but as you said, also expects great responsibility/discipline to cobble the proper properties together.
- PaulHoule 2y agoMathematically the boundaries of a record can be defined by production rules https://en.wikipedia.org/wiki/Business_rules_engine https://en.wikipedia.org/wiki/Business_rules_engine which could be written as SPARQL queries, I've used these to cut records out of a big graph, I haven't thought seriously if these could be built into a large scale general purpose systems. The most fun I ever had with Jena was when I used the rules engine for the control plane of a batch processing system which used stream processing primitives [1] https://jena.apache.org/documentation/inference/ https://jena.apache.org/documentation/inference/ The Jena folks said my use was completely unsupported, I had looked at the source code and got to understand how the rules engine worked and I knew damn well there was nothing wrong with what I was doing. I've thought a lot about why production rules have had so little impact on the industry, I mean people really hate drools https://www.drools.org/ https://www.drools.org/ That kind of system is particularly strong at handling deep asynchrony, like when a business process at a bank might involve some steps where you might have to wait for a loan office to approve a loan. It's disappointing to me that nobody has tried to use them (so far as I can tell) to deal with the asynchronous comms problems in Javascript though I've yet to get a clear picture in my mind about how to get started on that. (Funny I am getting an idea now so I'm putting a ticket on my personal Kanban board) [1] I worked later at a place that had a similar engine written in very awkward Scala that allegedly used Either and Optional for error handling but actually dropped errors most of the time; I knew what algebra my engine supported, they argued whether or not something like that had an algebra; my engine got the same answers every time because it tore down the system properly at the end, their engine gave different answers every time but they didn't seem to care
- smarx007 2y ago> Updating a SPARQL database is a little bit dangerous because there is no intrinsic idea of what a record is SPARQL has a notion of a transactional boundary just like SQL has. You can combine multiple SPARQL queries in one transaction, they will all succeed or all fail just like you'd expect.
- PaulHoule 2y agoSorta kinda. Your code has to put the right things in a transaction all the time for transactions for transactions to work right. If there is some flow of information like application does query -> application thinks -> application does update you have to wrap the whole sandwich in a transaction, people frequently don't do that. If I'm writing 20 of those for an application I want something that I know is bulletproof. My experience with SQL is that the average SQL developer doesn't really understand how to do transactions right but their ass gets saved (in a probabilistic sense) by the grouping of updates that is implicit by running an INSERT or an UPDATE against a table. There's also the fact that a lot of triple stores are seriously half baked research-quality code if that. Many triple stores struggle if you just try to load 100,000 triples sequentially, for an application like my YOShInOn RSS reader which I expect to use every day and not have to patch or maintain anything for 18+ months. (Ok, a 20GB database that needs to be pruned crept up on me gradually, but that's an arangodb problem, I'd expect the average triple to store to have crumbled 17 months ago.) I'd love to have something that updates like a document-oriented database but lets you run a SPARQL query against the union of all the documents. Database experts though always seem to change the subject when it comes to having a graph algebra that lets you UNION 10 million graphs. (For that matter, I sure as hell couldn't pitch any kind any kind of "boxes-and-lines" query tool [1] etc. that passed JSON documents/RDF graphs over the lines between the operators to the VCs and private equity people who were buying up query engines circa 2015 because they were hung up on the speed of columnar query engines... Despite the fact that the ones that pass relational rows over the lines require people who really aren't qualified to do so create analysis jobs that look like terrible hairballs because of all the joins they do.) [1] Alteryx, KNIME
- smarx007 2y ago