5 ms·

Heart: a planet-scale RDF data store

19 points by helwr 17y ago

mattyb 17y ago
Excited mark_l_watson?
- mark_l_watson 17y ago
  No. I look at this site occasionally, and nothing seems to get done. Also (pardon me if I take your good joke seriously :-) I don't see why we need planet scale RDF data stores. Instead, we need a very large number small SPARQL endpoints with some form of trust mechanism and a discovery mechanism.
  - mattyb 17y ago
    :-)
Raphael 17y ago
Skynet.
epochwolf 17y ago
So... someone is making a massive XML database out of some other components?
- blasdel 17y ago
  RDF is absolutely not XML. It's a directed graph, not a tree. It died primarily because the dimwits at the W3C saddled it with an atrocious XML serialization, but the semantic web bullshit didn't help, and neither did the poor use of it by Mozilla and others that led to it being used ineptly for RSS.
  - epochwolf 17y ago
    Thanks, I wasn't clear on what RDF was. I skimmed the wikipedia article and I thought it was just a variant of xml.
jerf 17y ago
I feel like this page is using "map-reduce" as magic juice. The problem described is enormous, and saying you're going to use map-reduce in this case is somewhat akin to saying you're going to be using CPUs. Yes, but so what? That doesn't magically move you to "feasible". I also get the sense that, far more than usual, this is technology without a use case. I'm not saying it's useless or that the idea is bad, just that I followed a couple of links and didn't find an even cursory discussion of what this is for or why it is better than other things, existing or otherwise. These aren't good signs.
- blasdel 17y ago
  It's a proposed original ASF project -- the point is to churn some bureaucracy, not produce something people want.
evgen 17y ago
Ouch. A directed graph in a columnar store like HBase? I guess there are worse approaches, but I would like to see a justification for this other than "we have this hammer called Hadoop..."
- blasdel 17y ago
  Got any better solutions? I don't. At least it ain't trying to do it relationally! Tuplestores are one of the only data stores I can think of that doesn't actively fight you when trying to query EAV data or traverse Directed Graphs.
  - epochwolf 17y ago
    If it's a directed graph, why not use a graph database?
    - blasdel 17y ago
      Tuplestores are one of the most common backend formats for storing graphs — you only need triples, or quads if you want timestamps. Everything's stored as (Entity, Attribute, Value) in the same table, with all of the Es and As being references, and with the Vs as references when the tuple is a graph edge but as atoms when it's a graph node. Pretty much every existing RDF database is a tuplestore underneath, or abuses a relational database as a crappy tuplestore that won't get you fired for installing it.
chime 17y ago
There is so much useful and well-organized information in RDF format out there that is completely being unused. Check out: http://wiki.dbpedia.org/Datasets http://wiki.dbpedia.org/Datasets - this includes almost the entire Wikipedia in a nice, organized way. Using Sparql/Snorql ( http://dbpedia.org/snorql/ http://dbpedia.org/snorql/ ) you can actually find everything that Bart Simpson wrote on the chalkboard during the opening sequence in season 12: http://www.snee.com/bobdc.blog/2007/11/querying-dbpedia.html http://www.snee.com/bobdc.blog/2007/11/querying-dbpedia.html However, searching through RDF is a pain and not a small task. The tools available to manage 50GB worth of RDF data are very basic and inefficient. I used a MySQL-based RDF store and even on my extremely powerful server, doing a simple query choked the machine. I'm hoping with a better data store, we can now manage RDF data better and hopefully search it better. Think of how much useful information there is on Wikipedia, Powerset.com etc. I don't mean just the text, but rather the extremely well-organized human-edited information boxes on the right that contain categories, hierarchical data, and specifications. Using RDF, all of that could be instantly queryable. I would love to write an app that queries a Google RDF store for local listings or map information. APIs are flat and tabular. I can only query Amazon for specific search words in items. Right now, I can't say with one query: "show me all ratings for every book that was written by the authors who currently have a book in the top 100 books list." RDF makes that possible. The problem right now is that RDF stores suck and searching through them isn't easy for the average developer. Here's hoping Apache makes it possible and scalable.
- jerf 17y ago
  "The problem right now is that RDF stores suck and searching through them isn't easy for the average developer." The question is, can there be a such thing as a non-sucky RDF store, though? RDF is a directed graph, essentially the single most general data structure possible. That means it can represent anything, awesome!... that also means, no optimizations that take advantage of the structure of the data are possible. It is not clear to me that the combination of "no effective optimizations" and "enormous internet-scale database" actually has a solution. (It certainly doesn't have an easy solution, you'd have to do something very sophisticated to extract structure back out and take advantage of it, while admitting the existence of arbitrary exceptions to your clever scheme. Even if possible, this is a recipe for an endless progression of the hardest sorts of bugs.)

19 points by helwr 17y ago

mattyb 17y ago

Excited mark_l_watson?

mark_l_watson 17y ago
No. I look at this site occasionally, and nothing seems to get done. Also (pardon me if I take your good joke seriously :-) I don't see why we need planet scale RDF data stores. Instead, we need a very large number small SPARQL endpoints with some form of trust mechanism and a discovery mechanism.
- mattyb 17y ago
  :-)

Raphael 17y ago

Skynet.

epochwolf 17y ago

So... someone is making a massive XML database out of some other components?

blasdel 17y ago
RDF is absolutely not XML. It's a directed graph, not a tree. It died primarily because the dimwits at the W3C saddled it with an atrocious XML serialization, but the semantic web bullshit didn't help, and neither did the poor use of it by Mozilla and others that led to it being used ineptly for RSS.
- epochwolf 17y ago
  Thanks, I wasn't clear on what RDF was. I skimmed the wikipedia article and I thought it was just a variant of xml.

jerf 17y ago

I feel like this page is using "map-reduce" as magic juice. The problem described is enormous, and saying you're going to use map-reduce in this case is somewhat akin to saying you're going to be using CPUs. Yes, but so what? That doesn't magically move you to "feasible". I also get the sense that, far more than usual, this is technology without a use case. I'm not saying it's useless or that the idea is bad, just that I followed a couple of links and didn't find an even cursory discussion of what this is for or why it is better than other things, existing or otherwise. These aren't good signs.

blasdel 17y ago
It's a proposed original ASF project -- the point is to churn some bureaucracy, not produce something people want.

evgen 17y ago

Ouch. A directed graph in a columnar store like HBase? I guess there are worse approaches, but I would like to see a justification for this other than "we have this hammer called Hadoop..."

blasdel 17y ago
Got any better solutions? I don't. At least it ain't trying to do it relationally! Tuplestores are one of the only data stores I can think of that doesn't actively fight you when trying to query EAV data or traverse Directed Graphs.
- epochwolf 17y ago
  If it's a directed graph, why not use a graph database?
  - blasdel 17y ago
    Tuplestores are one of the most common backend formats for storing graphs — you only need triples, or quads if you want timestamps. Everything's stored as (Entity, Attribute, Value) in the same table, with all of the Es and As being references, and with the Vs as references when the tuple is a graph edge but as atoms when it's a graph node. Pretty much every existing RDF database is a tuplestore underneath, or abuses a relational database as a crappy tuplestore that won't get you fired for installing it.

chime 17y ago

There is so much useful and well-organized information in RDF format out there that is completely being unused. Check out: http://wiki.dbpedia.org/Datasets http://wiki.dbpedia.org/Datasets - this includes almost the entire Wikipedia in a nice, organized way. Using Sparql/Snorql ( http://dbpedia.org/snorql/ http://dbpedia.org/snorql/ ) you can actually find everything that Bart Simpson wrote on the chalkboard during the opening sequence in season 12: http://www.snee.com/bobdc.blog/2007/11/querying-dbpedia.html http://www.snee.com/bobdc.blog/2007/11/querying-dbpedia.html However, searching through RDF is a pain and not a small task. The tools available to manage 50GB worth of RDF data are very basic and inefficient. I used a MySQL-based RDF store and even on my extremely powerful server, doing a simple query choked the machine. I'm hoping with a better data store, we can now manage RDF data better and hopefully search it better. Think of how much useful information there is on Wikipedia, Powerset.com etc. I don't mean just the text, but rather the extremely well-organized human-edited information boxes on the right that contain categories, hierarchical data, and specifications. Using RDF, all of that could be instantly queryable. I would love to write an app that queries a Google RDF store for local listings or map information. APIs are flat and tabular. I can only query Amazon for specific search words in items. Right now, I can't say with one query: "show me all ratings for every book that was written by the authors who currently have a book in the top 100 books list." RDF makes that possible. The problem right now is that RDF stores suck and searching through them isn't easy for the average developer. Here's hoping Apache makes it possible and scalable.

jerf 17y ago
"The problem right now is that RDF stores suck and searching through them isn't easy for the average developer." The question is, can there be a such thing as a non-sucky RDF store, though? RDF is a directed graph, essentially the single most general data structure possible. That means it can represent anything, awesome!... that also means, no optimizations that take advantage of the structure of the data are possible. It is not clear to me that the combination of "no effective optimizations" and "enormous internet-scale database" actually has a solution. (It certainly doesn't have an easy solution, you'd have to do something very sophisticated to extract structure back out and take advantage of it, while admitting the existence of arbitrary exceptions to your clever scheme. Even if possible, this is a recipe for an endless progression of the hardest sorts of bugs.)