4 ms·

I've been using RDF and triplestores / RDF databases for the last half-decade, developing both front-end and back-end systems, and training many developers to w

I've been using RDF and triplestores / RDF databases for the last half-decade, developing both front-end and back-end systems, and training many developers to work in RDF. If you're used to either relational databases or object-oriented design, it's a really different way of thinking about data. Just like OOP is really good for certain kinds of problems and models, and RDBMS is good for other kinds of problems and models, RDF is great for specific kinds of problems. For example, if you need to combine several data from several somewhat incongruent sources into a single coherent database (e.g. dozens of data feeds that are almost the same, but you need to preserve the differences while combining the parts that are the same), you might end up with headaches trying to come up with a good RDBMS design; RDF is really well-suited for that kind of problem. While a lot of the work I do is covered by NDA, one problem that I've applied it that I can talk about is analyzing basketball play-by-plays. I've spent some time talking to the analytics team at an NBA franchise, and it turns out doing interesting analytics on play-by-plays can be a surprisingly tough nut to crack. RDF was a great tool for tackling this. Here's the source (written in Scala), for anyone interested at having a look: https://github.com/andrewstellman/pbprdf https://github.com/andrewstellman/pbprdf

Jakob 9y ago
This is interesting, I always wanted to find a good use case for RDF but in the end RDBMS worked out fine. Sports events seem a good example. What made it easier for you in your example of basketball play-by-plays with RDF? Taking the first example https://github.com/andrewstellman/pbprdf#example-analyze-a-set-of-games https://github.com/andrewstellman/pbprdf#example-analyze-a-s... and translating it into an RDBMS approach seems rather straight forward: GameEvent (PersonA, EventType, PersonB, Game, Time) Roster (Person, Game, Team) To get the fouls drawn you then take SELECT Team, COUNT(Team) FROM GameEvent JOIN Roster ON GameEvent.Game = Roster.Game AND GameEvent.PersonB = Roster.Person WHERE GameEvent.EventType = "foul" GROUP BY Roster.Team Is it because of easier schema changes later on like introducing "secondsLeftInPeriod"? I suppose in a normalized relational scenario that could be something like: Period (Number, Game, StartTime, EndTime) And doing .. WHERE Period.EndTime - GameEvent.Time < 5 ..
- andrewstellman 9y ago
  > Is it because of easier schema changes later on like introducing "secondsLeftInPeriod"? That's definitely an important benefit. RDF makes it really easy to introduce changes that not only don't break the existing schema, but can be entirely isolated or combined in queries. Another thing that RDF makes easy is analysis that takes advantage of a graph -- using relationships with other players, shots, etc. What players have the highest percentage making 3-point shots in possessions immediately after a player on the other team missed a 3-point shot? Building queries you compare previous possessions, shots, quarters; players' relations to each other (e.g. performance players who were subbed in after previous teammates went scoreless for 3 possessions) -- these things are a lot easier to do in RDF than in SQL. Obviously, there are many things that are easier to model in RDBMS and query in SQL than with RDF/SPARQL. Every tool has its uses.
ar-jan 9y ago
Do you know if there's off-the-shelf software (GUI) to create/edit/explore your own RDF dataset? Or does it always involve building your own front-end?
- andrewstellman 9y ago
  I like WebVOWL for visualizing the RDF ontology. Here's the pbprdf ontology displayed in it: http://www.visualdataweb.de/webvowl/#iri=https://raw.githubusercontent.com/andrewstellman/pbprdf/master/generated/ontology.ttl http://www.visualdataweb.de/webvowl/#iri=https://raw.githubu... A few years ago I put together a quick GUI in C# to make it easier to run SPARQL queries: https://github.com/andrewstellman/sparql-explorer https://github.com/andrewstellman/sparql-explorer I haven't found an RDF editor or visual tool that I like. Some people like Topbraid Composer: https://www.topquadrant.com/tools/modeling-topbraid-composer-standard-edition/ https://www.topquadrant.com/tools/modeling-topbraid-composer... (commercial, closed source)
- MKolchin 9y ago
  Take a look at http://ontodia.org http://ontodia.org
- lolive 9y ago
  May be you can have a look at Datao: http://datao.net http://datao.net. PS: I am the maintainer, and we currently have some certificate issues. Drop me a mail to datao@datao.net if you want to be notified when the problem is solved.
conceptme 9y ago
Is there a particular RDF store which you would recommend? It seems scaling is a bit of an issue with Apache Jena. It's very easy to bring to entire system to a crawl with certain sparql queries and enough data in the store.
- andrewstellman 9y ago
  A team I've been working with for years has had a lot of success with Blazegraph: https://www.blazegraph.com/ https://www.blazegraph.com/ -- the marketing materials call it "ultra-scalable," and that actually turns out to be true. I haven't seen too many cases where specific queries will cause serious performance with the system. That said, we've done some work to prevent runaway queries (e.g. strict query timeouts, downstream systems that handle that situation gracefully).
  - mauimauimaui 9y ago
    I believe BlazeGraph is the basis for Amazon Neptune, the graph DB AWS announced at ReInvent last year. https://aws.amazon.com/neptune/ https://aws.amazon.com/neptune/
    - andrewstellman 9y ago
      That's great! I've had a chance to talk to some of the engineering and management folks at BlazeGraph over the years, and they're a really solid group. It's really nice to hear about a deserving team get success.
- WorldMaker 9y ago
  Semi-related, is there a good in-browser RDF store? Say something like what PouchDB does for CouchDB (and similar) JSON document stores. For as important as RDF has been for the web, it feels increasingly less "web native" today as most efforts still seem to be highly Java-focused and Browsers mostly don't run Java anymore. (I ask because a silly project idea I have some tiny amount of notes for is something of a Twine competitor. I realized that while the language ideas I'm exploring don't look like SPARQL or other graph languages, there is a bit of an overlap conceptually under the hood and an RDF store might make sense as a bootstrap tech, but brief searches didn't turn up anything useful.)
deleted 9y ago
[deleted]

Jakob 9y ago

This is interesting, I always wanted to find a good use case for RDF but in the end RDBMS worked out fine. Sports events seem a good example. What made it easier for you in your example of basketball play-by-plays with RDF? Taking the first example https://github.com/andrewstellman/pbprdf#example-analyze-a-set-of-games https://github.com/andrewstellman/pbprdf#example-analyze-a-s... and translating it into an RDBMS approach seems rather straight forward: GameEvent (PersonA, EventType, PersonB, Game, Time) Roster (Person, Game, Team) To get the fouls drawn you then take SELECT Team, COUNT(Team) FROM GameEvent JOIN Roster ON GameEvent.Game = Roster.Game AND GameEvent.PersonB = Roster.Person WHERE GameEvent.EventType = "foul" GROUP BY Roster.Team Is it because of easier schema changes later on like introducing "secondsLeftInPeriod"? I suppose in a normalized relational scenario that could be something like: Period (Number, Game, StartTime, EndTime) And doing .. WHERE Period.EndTime - GameEvent.Time < 5 ..

andrewstellman 9y ago
> Is it because of easier schema changes later on like introducing "secondsLeftInPeriod"? That's definitely an important benefit. RDF makes it really easy to introduce changes that not only don't break the existing schema, but can be entirely isolated or combined in queries. Another thing that RDF makes easy is analysis that takes advantage of a graph -- using relationships with other players, shots, etc. What players have the highest percentage making 3-point shots in possessions immediately after a player on the other team missed a 3-point shot? Building queries you compare previous possessions, shots, quarters; players' relations to each other (e.g. performance players who were subbed in after previous teammates went scoreless for 3 possessions) -- these things are a lot easier to do in RDF than in SQL. Obviously, there are many things that are easier to model in RDBMS and query in SQL than with RDF/SPARQL. Every tool has its uses.

ar-jan 9y ago

Do you know if there's off-the-shelf software (GUI) to create/edit/explore your own RDF dataset? Or does it always involve building your own front-end?

andrewstellman 9y ago
I like WebVOWL for visualizing the RDF ontology. Here's the pbprdf ontology displayed in it: http://www.visualdataweb.de/webvowl/#iri=https://raw.githubusercontent.com/andrewstellman/pbprdf/master/generated/ontology.ttl http://www.visualdataweb.de/webvowl/#iri=https://raw.githubu... A few years ago I put together a quick GUI in C# to make it easier to run SPARQL queries: https://github.com/andrewstellman/sparql-explorer https://github.com/andrewstellman/sparql-explorer I haven't found an RDF editor or visual tool that I like. Some people like Topbraid Composer: https://www.topquadrant.com/tools/modeling-topbraid-composer-standard-edition/ https://www.topquadrant.com/tools/modeling-topbraid-composer... (commercial, closed source)
MKolchin 9y ago
Take a look at http://ontodia.org http://ontodia.org
lolive 9y ago
May be you can have a look at Datao: http://datao.net http://datao.net. PS: I am the maintainer, and we currently have some certificate issues. Drop me a mail to datao@datao.net if you want to be notified when the problem is solved.

conceptme 9y ago

Is there a particular RDF store which you would recommend? It seems scaling is a bit of an issue with Apache Jena. It's very easy to bring to entire system to a crawl with certain sparql queries and enough data in the store.

andrewstellman 9y ago
A team I've been working with for years has had a lot of success with Blazegraph: https://www.blazegraph.com/ https://www.blazegraph.com/ -- the marketing materials call it "ultra-scalable," and that actually turns out to be true. I haven't seen too many cases where specific queries will cause serious performance with the system. That said, we've done some work to prevent runaway queries (e.g. strict query timeouts, downstream systems that handle that situation gracefully).
- mauimauimaui 9y ago
  I believe BlazeGraph is the basis for Amazon Neptune, the graph DB AWS announced at ReInvent last year. https://aws.amazon.com/neptune/ https://aws.amazon.com/neptune/
  - andrewstellman 9y ago
    That's great! I've had a chance to talk to some of the engineering and management folks at BlazeGraph over the years, and they're a really solid group. It's really nice to hear about a deserving team get success.
WorldMaker 9y ago
Semi-related, is there a good in-browser RDF store? Say something like what PouchDB does for CouchDB (and similar) JSON document stores. For as important as RDF has been for the web, it feels increasingly less "web native" today as most efforts still seem to be highly Java-focused and Browsers mostly don't run Java anymore. (I ask because a silly project idea I have some tiny amount of notes for is something of a Twine competitor. I realized that while the language ideas I'm exploring don't look like SPARQL or other graph languages, there is a bit of an overlap conceptually under the hood and an RDF store might make sense as a bootstrap tech, but brief searches didn't turn up anything useful.)

deleted 9y ago

[deleted]