4 ms·

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and i

by billpg 5y ago

(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.

mistrial9 5y ago
earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over
- udp 5y ago
  The ivory tower is working on it: https://github.com/w3c/EasierRDF https://github.com/w3c/EasierRDF
  - tasogare 5y ago
    RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
    - namedgraph 5y ago
      Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.
      - tasogare 5y ago
        Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format. And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.
        namedgraph 5y ago
        Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not. You're right about bioinformatics, but lets do a quick check on http://sparql.club/ http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.
    - udp 5y ago
      I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated. > Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.
  - liminal 5y ago
    I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.
- bawolff 5y ago
  Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that. In real life you use tools for both.
  - namedgraph 5y ago
    Triples are already structured, machine-readable data. HTML is not.
shock-value 5y ago
Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 https://www.wikidata.org/wiki/Q16156475).
- boxed 5y ago
  I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/ https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell. I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked. The code to do that is at https://github.com/boxed/relatedhow https://github.com/boxed/relatedhow
  - shock-value 5y ago
    There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever https://github.com/ad-freiburg/QLever) Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.
3pt14159 5y ago
Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew. When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.
- tcgv 5y ago
  > Don't worry about the haters To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite. But I know it's pretty common to see haters nitpicking things all around ;)
papercrane 5y ago
Hard to use it if you don't know about it! I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.
hutzlibu 5y ago
Yup, I had the same situation some months before, even though I knew wikidata was a thing. I know javascript and had the pages at hand. I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.

by billpg 5y ago

mistrial9 5y ago

earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over

udp 5y ago
The ivory tower is working on it: https://github.com/w3c/EasierRDF https://github.com/w3c/EasierRDF
- tasogare 5y ago
  RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
  - namedgraph 5y ago
    Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.
    - tasogare 5y ago
      Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format. And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.
      - namedgraph 5y ago
        Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not. You're right about bioinformatics, but lets do a quick check on http://sparql.club/ http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.
  - udp 5y ago
    I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated. > Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.
- liminal 5y ago
  I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.
bawolff 5y ago
Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that. In real life you use tools for both.
- namedgraph 5y ago
  Triples are already structured, machine-readable data. HTML is not.

shock-value 5y ago

Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 https://www.wikidata.org/wiki/Q16156475).

boxed 5y ago
I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/ https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell. I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked. The code to do that is at https://github.com/boxed/relatedhow https://github.com/boxed/relatedhow
- shock-value 5y ago
  There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever https://github.com/ad-freiburg/QLever) Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.

3pt14159 5y ago

Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew. When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.

tcgv 5y ago
> Don't worry about the haters To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite. But I know it's pretty common to see haters nitpicking things all around ;)

papercrane 5y ago

Hard to use it if you don't know about it! I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.

hutzlibu 5y ago

Yup, I had the same situation some months before, even though I knew wikidata was a thing. I know javascript and had the pages at hand. I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.