5 ms·

I was invited to work at Google in 2013 on an internal project using their Knowledge Graph. I had just written two books on RDF/SPARQL/linked-data and at first

I was invited to work at Google in 2013 on an internal project using their Knowledge Graph. I had just written two books on RDF/SPARQL/linked-data and at first I was taken aback by the query language used at the time. However, when I realized how well the KG scaled and how fast the queries were, I became a fan. BTW, I think this paper is very well written. Graph queries is not an easy topic and their examples and text make the data types, syntax, and how to perform matches and aggregations easy to follow.

ampdepolymerase 5y ago
What are your thoughts on just using plain datalog/prolog?
- mark_l_watson 5y ago
  I have only used datalog in the form of Clojure and Datomic on a consulting job about ten years ago. I used to be a huge Prolog fan, but I don't think that anyone has paid me to do Prolog develop in over 20 years. My largest Prolog project was porting a prototype AI planning system that I wrote in Common Lisp to ExperProlog. It took about 6 weeks to prototype the system in Common Lisp, and re-writing it in Prolog only took about two weeks. I noticed last month that someone was offering an online Prolog course using Swi-Prolog. That might be a good place to start.
- lmeyerov 5y ago
  Having written a chunk of my phd in prolog and even a simplified implementation of one, I'm definitely a fan. At the same time, our startup (graphistry) works w/ teams needing to look at all sorts of graph data, including knowledge graphs like the topic here, and I'd be nervous about meeting the performance & scale needs with out-of-the-box prolog systems. Likewise, a lot of prolog's power comes from ideas like automatically backtracking search, and I can't imagine using that in most team environments. Datalog solves most of that :) My read of the paper is their semantics largely reduces to datalog. The most interesting semantic exception is their bounded recursion, which suggests they're doing optimizations normal datalog wouldn't. Likewise, graph workloads often have weird phenomena to optimize for in terms of expected queries & data, where I'm guessing datalog might fit semantically but not how people would normally implement an engine. The PathQuery paper carefully side-stepped all such discussion, so I've been curious. At Graphistry, we do end-to-end GPU stuff, so I've been thinking about this particular problem, and came to a similar conclusion of a datalog-ish subset being the most straightforward way to get more performance/$ for this kind of task.
  - ampdepolymerase 5y ago
    You might like Google's Logica https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html https://opensource.googleblog.com/2021/04/logica-organizing-... (Datalog/Prolog family language compiled to SQL)
    - oddthink 5y ago
      I've been using this fairly heavily recently (internal to Google), to the point where I'm thinking of investing in writing an org-babel mode for it. It's a really nice way to structure queries!
      - lmeyerov 5y ago
        Oh wow that is neat! And yes, this kind of thing is why datalog is a lot more amenable to fast query plans & runtimes than prolog. This part is especially cool: https://github.com/EvgSkv/logica/blob/main/compiler/dialects.py https://github.com/EvgSkv/logica/blob/main/compiler/dialects...
jeffbee 5y ago
I'm interested in industry history, so I wonder if you can comment on how much of this, if any, came into Google from Metaweb. There are no Metaweb-connected people among the authors (that I can tell) but Warren Harris is acknowledged. Was it a language that was in development before the acquisition, or was it needed mainly after?
- remus404 5y ago
  As someone who worked on the Metaweb/KG team during that period, it's safe to say the development of PathQuery was a direct result of the acquisition. The lack of Metaweb-linked authors relates to the fact that PathQuery sits closer to 'serving knowledge' than 'representing knowledge'. But lessons learned from the development of MQL surely made their way into PathQuery, through Warren and others.
- dadkins 5y ago
  Mainly after. Metaweb brought MQL with them, which ran on their own single server graphd, which was eventually shutdown. Warren started working on PathQuery as a replacement for MQL. PathQuery was originally executed on Pregel and was far from a realtime query language. A realtime query engine was later developed for PathQuery in the search stack so interesting search queries could be answered a la minute from the Knowledge Graph, rather than just pre-generating results for a limited class of queries. This was all done in the 2012-2013 timeframe. Most of the people involved are long gone.
  - remus404 5y ago
    I've forgotten the details: by real-time query engine are you referring to DGraph, Livegraph or something else?
    - dadkins 5y ago
      Definitely not graphd. Maybe Livegraph? I forget what the project was called, but it was the first realtime query engine built for the Knowledge Graph and it used PathQuery as the query language.
criddell 5y ago
More than a decade ago I worked for a few years on a product that was based on OWL/RDF/SPARQL and since then I still see a lot of problems I'm facing in terms of subject, predicate, object triples. The semantic web feels like a fad that has clearly passed but there are plenty of good ideas from that field that are under-utilized. For example, a lot of what gets encoded as JSON can be described with these triples and once the data is in that form, there are interesting questions you can ask of it.
- darrenoc 5y ago
  I think it's far from a fad, the adoption of the semantic web has been enormous in the past 5 years. Almost every major tech company (and many financial institutions) now have their own Knowledge Graph team.

ampdepolymerase 5y ago

What are your thoughts on just using plain datalog/prolog?

mark_l_watson 5y ago
I have only used datalog in the form of Clojure and Datomic on a consulting job about ten years ago. I used to be a huge Prolog fan, but I don't think that anyone has paid me to do Prolog develop in over 20 years. My largest Prolog project was porting a prototype AI planning system that I wrote in Common Lisp to ExperProlog. It took about 6 weeks to prototype the system in Common Lisp, and re-writing it in Prolog only took about two weeks. I noticed last month that someone was offering an online Prolog course using Swi-Prolog. That might be a good place to start.
lmeyerov 5y ago
Having written a chunk of my phd in prolog and even a simplified implementation of one, I'm definitely a fan. At the same time, our startup (graphistry) works w/ teams needing to look at all sorts of graph data, including knowledge graphs like the topic here, and I'd be nervous about meeting the performance & scale needs with out-of-the-box prolog systems. Likewise, a lot of prolog's power comes from ideas like automatically backtracking search, and I can't imagine using that in most team environments. Datalog solves most of that :) My read of the paper is their semantics largely reduces to datalog. The most interesting semantic exception is their bounded recursion, which suggests they're doing optimizations normal datalog wouldn't. Likewise, graph workloads often have weird phenomena to optimize for in terms of expected queries & data, where I'm guessing datalog might fit semantically but not how people would normally implement an engine. The PathQuery paper carefully side-stepped all such discussion, so I've been curious. At Graphistry, we do end-to-end GPU stuff, so I've been thinking about this particular problem, and came to a similar conclusion of a datalog-ish subset being the most straightforward way to get more performance/$ for this kind of task.
- ampdepolymerase 5y ago
  You might like Google's Logica https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html https://opensource.googleblog.com/2021/04/logica-organizing-... (Datalog/Prolog family language compiled to SQL)
  - oddthink 5y ago
    I've been using this fairly heavily recently (internal to Google), to the point where I'm thinking of investing in writing an org-babel mode for it. It's a really nice way to structure queries!
    - lmeyerov 5y ago
      Oh wow that is neat! And yes, this kind of thing is why datalog is a lot more amenable to fast query plans & runtimes than prolog. This part is especially cool: https://github.com/EvgSkv/logica/blob/main/compiler/dialects.py https://github.com/EvgSkv/logica/blob/main/compiler/dialects...

jeffbee 5y ago

I'm interested in industry history, so I wonder if you can comment on how much of this, if any, came into Google from Metaweb. There are no Metaweb-connected people among the authors (that I can tell) but Warren Harris is acknowledged. Was it a language that was in development before the acquisition, or was it needed mainly after?

remus404 5y ago
As someone who worked on the Metaweb/KG team during that period, it's safe to say the development of PathQuery was a direct result of the acquisition. The lack of Metaweb-linked authors relates to the fact that PathQuery sits closer to 'serving knowledge' than 'representing knowledge'. But lessons learned from the development of MQL surely made their way into PathQuery, through Warren and others.
dadkins 5y ago
Mainly after. Metaweb brought MQL with them, which ran on their own single server graphd, which was eventually shutdown. Warren started working on PathQuery as a replacement for MQL. PathQuery was originally executed on Pregel and was far from a realtime query language. A realtime query engine was later developed for PathQuery in the search stack so interesting search queries could be answered a la minute from the Knowledge Graph, rather than just pre-generating results for a limited class of queries. This was all done in the 2012-2013 timeframe. Most of the people involved are long gone.
- remus404 5y ago
  I've forgotten the details: by real-time query engine are you referring to DGraph, Livegraph or something else?
  - dadkins 5y ago
    Definitely not graphd. Maybe Livegraph? I forget what the project was called, but it was the first realtime query engine built for the Knowledge Graph and it used PathQuery as the query language.

criddell 5y ago

More than a decade ago I worked for a few years on a product that was based on OWL/RDF/SPARQL and since then I still see a lot of problems I'm facing in terms of subject, predicate, object triples. The semantic web feels like a fad that has clearly passed but there are plenty of good ideas from that field that are under-utilized. For example, a lot of what gets encoded as JSON can be described with these triples and once the data is in that form, there are interesting questions you can ask of it.

darrenoc 5y ago
I think it's far from a fad, the adoption of the semantic web has been enormous in the past 5 years. Almost every major tech company (and many financial institutions) now have their own Knowledge Graph team.