11 ms·

SPARQL over Wikidata is one of the closest things to magic on the Internet. It's incomplete, the runtime is slow, it dies on too much data, but it's amazing bei

by thom 5y ago

SPARQL over Wikidata is one of the closest things to magic on the Internet. It's incomplete, the runtime is slow, it dies on too much data, but it's amazing being able to think up random questions and actually ask them, however frivolous. Stuff like "who's the most successful footballer from my home town?" or "which part of India has produced the most test cricketers?" is really easy. Makes me wish I'd taken the quantified self more seriously because there's so much I'd love to find out with rich enough linked data.

zozbot234 5y ago
You can download a data dump and run your own queries locally if the limitations of the online service are of concern. Full dumps of the whole Wikidata are easily available, or you can export a limited result set from the query service and run your own local queries on that data.
arp242 5y ago
I found SPARQL pretty hard to deal with. Some time ago I wanted to make an overview of various election results to prove a certain point. The data is in there, I can see it in the UI, but I had a lot of trouble writing a script to retrieve it (so I could make a nice table and/or chart). Basically, after an hour I failed and gave up. I will readily admit this is due to my own ignorance; I just wanted to post a point in a Reddit comment and investing a lot of time to properly learn SPARQL wasn't worth the effort, but I wish the data was more easily accessible without a fairly large time investment. WikiData is basically the only structured source for these election results that I could find (the government publishes them of course, but in a PDF that's always different). And I'm a programmer by trade! Non-technical people will have an even harder time. WikiData is pretty good, but I feel it could be truly fantastic if the data was more easily accessible.
- zozbot234 5y ago
  You have to be familiar with Wikidata's underlying model and how the data you care about are mapped to it, but having done that it's just a matter of picking one of the many example queries and tweaking it so it references the right bits of data.
- thom 5y ago
  Yeah, happy to admit SPARQL itself is hard, and every query requires having five wikidata tabs and three stackoverflow ones open. But, I find myself battling through because the results are worth it.
  - kzrdude 5y ago
    How do you deal with incomplete data? I made a script to get the highest mountains in norway and get them on a map in Jupyter - looks nice, includes pictures if there are etc, but it doesn't pick out all the highest mountains. Most of them, but some are missing.
    - thom 5y ago
      I will admit I rarely use it for things that are so mission critical that this would matter. That said, it's always nice to leave the data more complete than you found it in these cases.
- bawolff 5y ago
  I think a big reason that sparql is hard, is it looks like SQL but has a really different data model. People who know sql tend to apply sql intuitions to sparql, which leads to confusion. But once you get past the learning curve its really not that bad.
  - arp242 5y ago
    > once you get past the learning curve its really not that bad. Kind of applies to everything, no? :-) But yeah, I don't think GraphQL is "bad", it's just hard to get started with (same applies to SQL really, although we're all used to that so we've forgotten about getting started with it). I want to use WikiData maybe ... once a year? Or less? I will probably have forgotten a lot of stuff next time I want to use it.

by thom 5y ago

zozbot234 5y ago

You can download a data dump and run your own queries locally if the limitations of the online service are of concern. Full dumps of the whole Wikidata are easily available, or you can export a limited result set from the query service and run your own local queries on that data.

arp242 5y ago

I found SPARQL pretty hard to deal with. Some time ago I wanted to make an overview of various election results to prove a certain point. The data is in there, I can see it in the UI, but I had a lot of trouble writing a script to retrieve it (so I could make a nice table and/or chart). Basically, after an hour I failed and gave up. I will readily admit this is due to my own ignorance; I just wanted to post a point in a Reddit comment and investing a lot of time to properly learn SPARQL wasn't worth the effort, but I wish the data was more easily accessible without a fairly large time investment. WikiData is basically the only structured source for these election results that I could find (the government publishes them of course, but in a PDF that's always different). And I'm a programmer by trade! Non-technical people will have an even harder time. WikiData is pretty good, but I feel it could be truly fantastic if the data was more easily accessible.

zozbot234 5y ago
You have to be familiar with Wikidata's underlying model and how the data you care about are mapped to it, but having done that it's just a matter of picking one of the many example queries and tweaking it so it references the right bits of data.
thom 5y ago
Yeah, happy to admit SPARQL itself is hard, and every query requires having five wikidata tabs and three stackoverflow ones open. But, I find myself battling through because the results are worth it.
- kzrdude 5y ago
  How do you deal with incomplete data? I made a script to get the highest mountains in norway and get them on a map in Jupyter - looks nice, includes pictures if there are etc, but it doesn't pick out all the highest mountains. Most of them, but some are missing.
  - thom 5y ago
    I will admit I rarely use it for things that are so mission critical that this would matter. That said, it's always nice to leave the data more complete than you found it in these cases.
bawolff 5y ago
I think a big reason that sparql is hard, is it looks like SQL but has a really different data model. People who know sql tend to apply sql intuitions to sparql, which leads to confusion. But once you get past the learning curve its really not that bad.
- arp242 5y ago
  > once you get past the learning curve its really not that bad. Kind of applies to everything, no? :-) But yeah, I don't think GraphQL is "bad", it's just hard to get started with (same applies to SQL really, although we're all used to that so we've forgotten about getting started with it). I want to use WikiData maybe ... once a year? Or less? I will probably have forgotten a lot of stuff next time I want to use it.