11 ms·

Wikidata or Scraping Wikipedia

178 points by Lockal 5y ago

JamesCoyne 5y ago
https://web.archive.org/web/20210822151254/http://simia.net/wiki/Wikidata_or_scraping_Wikipedia https://web.archive.org/web/20210822151254/http://simia.net/...
dhosek 5y ago
On a related point, while doing some Unicode research, I discovered that the Unicode project itself uses wikidata as an (untrusted) source for some data, translations of names, if I recall correctly, cf. https://www.unicode.org/review/pri408/pri408-tr51-QID.html https://www.unicode.org/review/pri408/pri408-tr51-QID.html although that's not the reference I encountered earlier today. Their system is set up so that if the Unicode organization corrects something previously read, it takes precedence over what was pulled from wikidata, but otherwise the wikidata value will be used.
- dhosek 5y ago
  Ah, this is what I read earlier today, yay Google+color-changing links http://cldr.unicode.org/implementers-faq http://cldr.unicode.org/implementers-faq
exogen 5y ago
I missed the original HN and twitter threads referenced in the post, so I might just be repeating something that was already said there... But, in nearly all cases I would trust a bespoke Wikipedia scraper over using the output of Wikidata or DBpedia. Not to disparage either project, because they're great ideas and good efforts. I have a firm grasp of RDF and SPARQL queries (used to work with them professionally), which also makes them tempting to use. One issue is that Wikidata tends to only report facts whose subjects or objects themselves have articles (and thus Wikidata entities). For example, compare the "Track listing" section of Carly Rae Jepsen's Curiosity EP on Wikipedia vs. the "track listing" property on Wikidata. Wikipedia has: 1. Call Me Maybe (link) 2. Curiosity (link) 3. Picture 4. Talk to Me 5. Just a Step Away 6. Both Sides Now (link) while Wikidata has: 1. Call Me Maybe 2. Curiosity So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now"). > Others asked about the data quality of Wikidata, and complained about the huge amount of bad data, duplicates, and the bad ontology in Wikidata (as if Wikipedia wouldn’t have these problems. I mean how do you figure out what a Wikipedia article is about? How do you get a list of all bridges or events from Wikipedia?) Often the problem isn't that Wikipedia is wrong, it's that Wikidata's own parser (however it works) doesn't account for the many ways people format things on Wikipedia. With a bespoke parser, you can improve it over time as you encounter edge cases. With Wikidata, you can't really fix anything... the data is already extracted (right or wrong) and all the original context lost.
- valexiev 5y ago
  Scientific Articles are in a similar situation: when importing one from a bibliography database, you won't always find every author... So people made an alternative prop "author name" and some disambiguation tools that allow users to gradually replace those with "author" links to real persons. Let's put aside the question whether each song ever written should be in WD: I believe all of this data, modeled more elaborately, is available on MusicBrainz. There's a difference between a work (eg "Both Sides Now") and its particular rendition in an album (as you said that track is "a cover"), and MusicBrainz makes that distinction and captures both, but I think WD doesn't (I don't work on music in WD, so I haven't checked). If you really want all this data in WD then I guess you could import it from MusicBrainz... a massive undertaking. > Wikidata's own parser (however it works) There's no such thing (in contrast, DBpedia has the dbpedia extraction framework, which is fairly good but not perfect and suffers greatly from the various ways people use to describe the same thing). WD has tools like QS and wikibase-cli, and people write bots to scrape and contribute specific kinds of data. BTW the Wikipedia link https://en.wikipedia.org/wiki/Both_Sides,_Now#Other_recordings#2010s https://en.wikipedia.org/wiki/Both_Sides,_Now#Other_recordin... is wrong (invalid): you can't have two anchors in a link. Cheers!
- mistrial9 5y ago
  acute observation! there is something here to be teased out .. about.. the final product is a human readable page all these years, and that human readable page got better in adhoc ways and most all of those improvments stuck.. compare to the RDF efforts, who ride a rigorous math-y perspective and with a far, far smaller development crowd right away..
- jancsika 5y ago
  > So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now"). In other words, "scraping wikipedia" is the answer to the question implied in the HN title to this post. :)
  - vrandecic 5y ago
    I'd suggest that in this case one should consider using MusicBrainz, in order to get more comprehensive and better results than either with Wikidata or Wikipedia.
svat 5y ago
This is a great post, which also happens to serve as a good illustration of the "curse of knowledge" and the typical blind-spots of enthusiasts. Consider the timeline of events: • The blog post on scraping Wikipedia (https://billpg.com/data-mining-wikipedia/ https://billpg.com/data-mining-wikipedia/ , HN discussion 4 days ago: https://news.ycombinator.com/item?id=28234122 https://news.ycombinator.com/item?id=28234122 which mentions Wikidata as an alternative etc. • The author of this post, a Wikidata person, finds this an "extremely surprising discussion", and posts a Twitter thread ( https://web.archive.org/web/20210820105621/https://twitter.com/vrandezo/status/1428501116401061899 https://web.archive.org/web/20210820105621/https://twitter.c... ) ending with > I don't want to argue or disagree, I am just completely surprised by that statement. Are the docs so bad? Is the API design of Wikidata so weird or undiscoverable? There are plenty of libraries for getting Wikidata data, are they all so hard to use? I am really curious. This curiosity is a great attitude! (But…) • After seeing the HN discussion and responses on Twitter/Facebook, he writes this post linked here. In this post, he does mention what he learned from potential users: > And there were some very interesting stories about the pain of using Wikidata, and I very much expect us to learn from them and hopefully make things easier. The number of API queries one has to make in order to get data […], the learning curve about SPARQL and RDF (although, you can ignore both, unless you want to use them explicitly - you can just use JSON and the Wikidata API), the opaqueness of the identifiers (wdt:P25 wd:Q9682 instead of “mother” and “Queen Elizabeth II”) were just a few. The documentation seems hard to find, there seem to be a lack of libraries and APIs that are easy to use. And yet, comments like "if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you" surprised me a lot. […] I am not here to fight. I am here to listen and to learn, in order to help figuring out what needs to be made better. Again, very commendable! Almost an opening to really understanding the perspective of casual potential users. But then: the entire rest of the post does not really address "the other side", and instead completely focuses on the kinds of things Wikidata enthusiasts care about: comparing Wikipedia and Wikidata quality in this example, etc. I mean, sure, this query he presents is short: select * { wd:Q9682 (wdt:P25|wdt:P22)* ?p . ?p wdt:P25|wdt:P22 ?q } but when he says: > I would claim that I invested far less work than Bill in creating my graph data. No data cleansing, no scraping, no crawling, no entity reconciliation, no manual checking. he's ignoring the work he invested in learning that query language (and where to query it), for instance. And this post would have been a perfect opportunity to teach readers about how to go from the question "all ancestors of Queen Elizabeth" to that query (and in trying to teach it, he may have better discovered exactly what is hard about it), but he just squanders the opportunity (just as when he says "plenty of libraries" without inviting exploration by linking to the easiest one): this is a typical thing enthusiasts do, which is unfortunate IMO. When scraping HTML from Wikipedia, one is using general-purpose well-known tools. You'll get slightly better at whatever general-purpose programming language and libraries you were using, learn something that may be useful the next time you need to scrape something else. And most importantly, you know that you'll finish, you can see a path to success. When exploring something "alternative" like Wikidata, you aren't sure if it will work, so the alternative path needs to work harder to convince potential users of success. --- Personal story: I actually know about the existence of Wikidata. Yet the one time I tried to use it, I couldn't figure out how. This is what I was trying to do: plot a graph of the average age of Turing Award winners by year. (Reproduce the first figure from here: http://hagiograffiti.blogspot.com/2009/01/when-will-singularity-occur.html http://hagiograffiti.blogspot.com/2009/01/when-will-singular... just for fun) One would think this is a perfect use-case for Wikidata: presumably it has a way of going from Turing Award → list of winners → each winner's date of birth. But I was stymied at the very first step: despite knowing of the existence of Wikidata, and being able to go from the Wikipedia page that lists all recipients (current version: https://en.wikipedia.org/w/index.php?title=Turing_Award&oldid=1028806461#Recipients https://en.wikipedia.org/w/index.php?title=Turing_Award&oldi... ) to the Wikidata item for "Turing Award" (look for "Wikidata item" in the sidebar on the left) https://www.wikidata.org/wiki/Q185667 https://www.wikidata.org/wiki/Q185667 I could not quickly find a way of getting a list of recipients from there. Tantalizingly, the data maybe does exist e.g. if I go to one of the recipients like Leslie Valiant https://www.wikidata.org/wiki/Q93154 https://www.wikidata.org/wiki/Q93154 I see a "statement" award received → Turing Award with "property" point in time → 2010. Even after coming so close, and being interested in using Wikidata, it was not easy enough for me to get to the next step (which I still imagine is possible, maybe with tens of minutes of effort), until I just decided "screw this, I'll just scrape the Wikipedia page" (I scraped the wikisource rather than html). And if one is going to have to scrape anyway, then might as well do the rest too (dates of birth) with scraping.
nine_k 5y ago
I think this project is not getting enough attention: https://github.com/zverok/wikipedia_ql https://github.com/zverok/wikipedia_ql It allows to query Wikipedia (not wikidata, but the actual human-readable text) more or less directly, mixing the way you describe a scraper with some nicer higher-level constructs. Can't vouch for its performance, but the API is interesting and nice.
- aasasd 5y ago
  My personal long-standing wish is querying categories, in which the pages have the same infoboxes, by the fields in the boxes. Preferably without waiting to download dozens or hundreds of pages first. The infobox→Wikidata integration would pretty much solve that (not the other way around), and I'm told that the Wikidata Bridge project aims to do that integration: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge However, if someone made a different database that would be queryable tomorrow, I wouldn't mind.
  - valexiev 5y ago
    Use https://www.mediawiki.org/wiki/Wikidata_Query_Service/Categories https://www.mediawiki.org/wiki/Wikidata_Query_Service/Catego... by making a federated query to https://query.wikidata.org/bigdata/namespace/categories/sparql https://query.wikidata.org/bigdata/namespace/categories/spar.... Cheers!
    - aasasd 5y ago
      Thanks, but this sounds like it only says what subcategories and, perhaps, pages are in the categories—but doesn't contain any data from the pages themselves. My main target is kinda-structured data from infoboxes—e.g. genre, platform, year for videogames. I don't even need categories particularly—I just grab all pages from them, hoping that all the pages I would want are in these categories.
  - vrandecic 5y ago
    Sounds a lot like DBpedia to me
ComputerGuru 5y ago
Site is down. Archive.org link: https://web.archive.org/web/20210821004307/http://simia.net/wiki/Wikidata_or_scraping_Wikipedia https://web.archive.org/web/20210821004307/http://simia.net/...
simonw 5y ago
Here-in lies the problem for me: select * { wd:Q9682 (wdt:P25|wdt:P22)* ?p . ?p wdt:P25|wdt:P22 ?q } - I am extremely motivated to learn how to use this: I have a deep desire to extract data from Wikipedia, and I'm fascinated by graph databases. And yet, despite trying on several previous occasions, SPARQL has completely failed to stick in my brain. This is partly my own failing: I'm confident that if I really dedicated myself to it I could get over this hump. But it's also a sign that the learning curve here really is tremendously steep, which I think indicates a problem with the design of the technology.
- mmarx 5y ago
  A few days ago, the Wikidata Query Builder[0] was deployed. It provides a visual interface to generate simple SPARQL queries, and you can show the generated queries. Maybe this can help you in understanding how SPARQL patterns work? [0] https://query.wikidata.org/querybuilder/ https://query.wikidata.org/querybuilder/
  - simonw 5y ago
    That does look like a big step forward. It could really benefit from some linked examples on that page though - I stared at the interface for quite a while, unable to figure out how to use it for anything - then I dug around for an example link and it started to make sense to me. https://query.wikidata.org/querybuilder/?query=%7B%22conditions%22%3A%5B%7B%22propertyId%22%3A%22P1435%22%2C%22propertyDataType%22%3A%22wikibase-item%22%2C%22propertyValueRelation%22%3A%22matching%22%2C%22referenceRelation%22%3A%22regardless%22%2C%22value%22%3A%22Q10729054%22%2C%22subclasses%22%3Afalse%2C%22conditionRelation%22%3Anull%2C%22negate%22%3Afalse%7D%2C%7B%22propertyId%22%3A%22P131%22%2C%22propertyDataType%22%3A%22wikibase-item%22%2C%22propertyValueRelation%22%3A%22regardless-of-value%22%2C%22referenceRelation%22%3A%22regardless%22%2C%22value%22%3A%22%22%2C%22subclasses%22%3Atrue%2C%22conditionRelation%22%3A%22and%22%2C%22negate%22%3Afalse%7D%5D%2C%22limit%22%3A100%2C%22useLimit%22%3Atrue%2C%22omitLabels%22%3Afalse%7D https://query.wikidata.org/querybuilder/?query=%7B%22conditi...
- dmitriid 5y ago
  Or use something like https://github.com/zverok/wikipedia_ql https://github.com/zverok/wikipedia_ql that uses Mediawiki API
nathell 5y ago
Wikipedia asks people not to crawl it. There are database dumps that you can instead import into your local MySQL and work from there. https://en.wikipedia.org/wiki/Wikipedia:Database_download#Please_do_not_use_a_web_crawler https://en.wikipedia.org/wiki/Wikipedia:Database_download#Pl...
- bawolff 5y ago
  Wikipedia has no objection to crawling a couple thousand pages if you do so at a reasonable speed and set a user-agent with a contact email. If you want to crawl millions of pages please use a dump.
elcapitan 5y ago
How do other people use Wikidata dumps if they are not using the "official" (with sparql or so) way of querying it? I have done some pretty raw extraction from it (e.g. download the already pretty large zipped json dump, then unzip it on the fly and parse the json, and extract triples and entities). Not sure if that is really quite efficient, but the dumps are hard to work with, and I really just needed the entities in one language and the triples/graph of them.
- brokensegue 5y ago
  They release a json dump but also just a dump of true triples in nt format (basically just tsv). It's few tens of gigs uncompressed
- IAmEveryone 5y ago
  This tool https://wdumps.toolforge.org/ https://wdumps.toolforge.org/ allows you to create bespoke dumps that are pre-filtered for your needs.
drej 5y ago
It’s fairly difficult from the other side as well - contributing. I’ve been trying to complete wikidata from a few open source datasets I am intensly familiar with… and it’s been rather painful. WD is the sole place I have ever interacted with that uses RDF, so I always forget the little syntax I learned last time around. I have some pre-existing queries versioned, because I’ll never be able to write them again. I even went to a local Wikimedia training to get acquainted with some necessary tooling, but I’m still super unproductive compared to e.g. SQL. It’s sad, really, I’d love to contribute more, but the whole data model is so clunky to work with.
- valexiev 5y ago
  I have no problems with the data model, but sadly you can't insert RDF statements: you have to go through tools like QS and wikidata-cli and the WD update performance is dismal. See https://phabricator.wikimedia.org/T290061 https://phabricator.wikimedia.org/T290061, which I posted in https://phabricator.wikimedia.org/project/board/5504/ https://phabricator.wikimedia.org/project/board/5504/ for DataQualityDays2021
- drej 5y ago
  That being said, I now remember I stopped contributing for a slightly different reason. While I tried to fill WD with complete information about a given subject, this was never leverage by a Wikimedia project - there is certain resistance to generating Wikipedia articles/infoboxes from Wikidata, so you're fighting on two fronts, you always have to edit things in two places and it's just a waste of everyone's time. Unless the attitude becomes "all facts in infoboxes and most tables come from WD", the two "datasets" will continue diverging. That is obviously more easily said than done, because relying on WD makes Wikipedia contribution a lot more difficult... and that pretty much defeats its purpose.
  - dane-pgp 5y ago
    > the two "datasets" will continue diverging. You may be pleased to learn that there is a project underway that aims to largely solve that problem: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge The last piece of news I can immediately find is that it was deployed to the Catalan Wikipedia in August 2020, but I'm not sure what progress there has been since.
dheera 5y ago
I'd highly recommend parsing the mobile edition of Wikipedia, it is much easier to parse.
marto1 5y ago
For me, I ended up using both. Opting for wikidata wherever made sense, but a lot of things felt half built/broken.
bob229 5y ago
Wikipedia is garbage.
aasasd 5y ago
I'm all for Wikidata, it's great in some ‘high-profile’ cases like data on countries—at least by my moderate standards. I didn't have much problem with Sparql, or perhaps my queries were simple. However, once you get into the lowbrow territory of e.g. modern cultural artifacts, people just edit Wikipedia way more, end of story. Want to know what games of some genre were made for platforms of your choice, sorted by year? You go to Wikipedia, not Wikidata. I'm told that there's a project to integrate infoboxes with Wikidata, so that their info goes into Wikidata when edited (not the other way around)—which would solve a large part of this scarcity, if the integration is seamless enough. Haven't yet seen it in action. Here's the project, Wikidata Bridge: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge
aaron695 5y ago
Wikidata has failed. This is not their fault, we have know for 10+ years machine readable data fails. See tags, and their failure (hashtag != tag) Their example of a goat, a simple common animal, does not identify how many teats a goat has ( does it? ) https://www.wikidata.org/wiki/Q2934 https://www.wikidata.org/wiki/Q2934 https://en.wikipedia.org/wiki/Goat https://en.wikipedia.org/wiki/Goat Everyone starting does family trees. Because they are easy and easily defined. But even this articles query ends up at fictional characters. The "period of lactation" for a goat is not a number. And it's not even one graph. It's multiple graphs which we don't have the data to accurately know. The original article was 100% correct. Web-scraping was the way to get the data. Web-scraping is a very useful and transferable skill. There's no point learning skills on a known failed idea like machine readable data. Wikipedia allows web scraping, anyone who tells you different is lying, see their robots.txt to make sure you don't get rate limited if doing massive amounts https://en.wikipedia.org/robots.txt https://en.wikipedia.org/robots.txt (and to see the stuff they don't want you to read). They also have downloadable dumps you can use.
shp0ngle 5y ago
for me, as an occasional wikipedia editor, wikidata is just the annoying thing I need to deal with when linking different language wikis together. I can never figure out anything there, but it’s just the UX I guess
denton-scratch 5y ago
My attention was caught by "Albert The Bear" (second list). It reminded me of Albert And The Lion: https://www.youtube.com/watch?v=oaw-savyK0s https://www.youtube.com/watch?v=oaw-savyK0s

178 points by Lockal 5y ago

https://web.archive.org/web/20210822151254/http://simia.net/wiki/Wikidata_or_scraping_Wikipedia https://web.archive.org/web/20210822151254/http://simia.net/...

dhosek 5y ago

On a related point, while doing some Unicode research, I discovered that the Unicode project itself uses wikidata as an (untrusted) source for some data, translations of names, if I recall correctly, cf. https://www.unicode.org/review/pri408/pri408-tr51-QID.html https://www.unicode.org/review/pri408/pri408-tr51-QID.html although that's not the reference I encountered earlier today. Their system is set up so that if the Unicode organization corrects something previously read, it takes precedence over what was pulled from wikidata, but otherwise the wikidata value will be used.

dhosek 5y ago
Ah, this is what I read earlier today, yay Google+color-changing links http://cldr.unicode.org/implementers-faq http://cldr.unicode.org/implementers-faq

exogen 5y ago

I missed the original HN and twitter threads referenced in the post, so I might just be repeating something that was already said there... But, in nearly all cases I would trust a bespoke Wikipedia scraper over using the output of Wikidata or DBpedia. Not to disparage either project, because they're great ideas and good efforts. I have a firm grasp of RDF and SPARQL queries (used to work with them professionally), which also makes them tempting to use. One issue is that Wikidata tends to only report facts whose subjects or objects themselves have articles (and thus Wikidata entities). For example, compare the "Track listing" section of Carly Rae Jepsen's Curiosity EP on Wikipedia vs. the "track listing" property on Wikidata. Wikipedia has: 1. Call Me Maybe (link) 2. Curiosity (link) 3. Picture 4. Talk to Me 5. Just a Step Away 6. Both Sides Now (link) while Wikidata has: 1. Call Me Maybe 2. Curiosity So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now"). > Others asked about the data quality of Wikidata, and complained about the huge amount of bad data, duplicates, and the bad ontology in Wikidata (as if Wikipedia wouldn’t have these problems. I mean how do you figure out what a Wikipedia article is about? How do you get a list of all bridges or events from Wikipedia?) Often the problem isn't that Wikipedia is wrong, it's that Wikidata's own parser (however it works) doesn't account for the many ways people format things on Wikipedia. With a bespoke parser, you can improve it over time as you encounter edge cases. With Wikidata, you can't really fix anything... the data is already extracted (right or wrong) and all the original context lost.

valexiev 5y ago
Scientific Articles are in a similar situation: when importing one from a bibliography database, you won't always find every author... So people made an alternative prop "author name" and some disambiguation tools that allow users to gradually replace those with "author" links to real persons. Let's put aside the question whether each song ever written should be in WD: I believe all of this data, modeled more elaborately, is available on MusicBrainz. There's a difference between a work (eg "Both Sides Now") and its particular rendition in an album (as you said that track is "a cover"), and MusicBrainz makes that distinction and captures both, but I think WD doesn't (I don't work on music in WD, so I haven't checked). If you really want all this data in WD then I guess you could import it from MusicBrainz... a massive undertaking. > Wikidata's own parser (however it works) There's no such thing (in contrast, DBpedia has the dbpedia extraction framework, which is fairly good but not perfect and suffers greatly from the various ways people use to describe the same thing). WD has tools like QS and wikibase-cli, and people write bots to scrape and contribute specific kinds of data. BTW the Wikipedia link https://en.wikipedia.org/wiki/Both_Sides,_Now#Other_recordings#2010s https://en.wikipedia.org/wiki/Both_Sides,_Now#Other_recordin... is wrong (invalid): you can't have two anchors in a link. Cheers!
mistrial9 5y ago
acute observation! there is something here to be teased out .. about.. the final product is a human readable page all these years, and that human readable page got better in adhoc ways and most all of those improvments stuck.. compare to the RDF efforts, who ride a rigorous math-y perspective and with a far, far smaller development crowd right away..
jancsika 5y ago
> So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now"). In other words, "scraping wikipedia" is the answer to the question implied in the HN title to this post. :)
- vrandecic 5y ago
  I'd suggest that in this case one should consider using MusicBrainz, in order to get more comprehensive and better results than either with Wikidata or Wikipedia.

svat 5y ago

This is a great post, which also happens to serve as a good illustration of the "curse of knowledge" and the typical blind-spots of enthusiasts. Consider the timeline of events: • The blog post on scraping Wikipedia (https://billpg.com/data-mining-wikipedia/ https://billpg.com/data-mining-wikipedia/ , HN discussion 4 days ago: https://news.ycombinator.com/item?id=28234122 https://news.ycombinator.com/item?id=28234122 which mentions Wikidata as an alternative etc. • The author of this post, a Wikidata person, finds this an "extremely surprising discussion", and posts a Twitter thread ( https://web.archive.org/web/20210820105621/https://twitter.com/vrandezo/status/1428501116401061899 https://web.archive.org/web/20210820105621/https://twitter.c... ) ending with > I don't want to argue or disagree, I am just completely surprised by that statement. Are the docs so bad? Is the API design of Wikidata so weird or undiscoverable? There are plenty of libraries for getting Wikidata data, are they all so hard to use? I am really curious. This curiosity is a great attitude! (But…) • After seeing the HN discussion and responses on Twitter/Facebook, he writes this post linked here. In this post, he does mention what he learned from potential users: > And there were some very interesting stories about the pain of using Wikidata, and I very much expect us to learn from them and hopefully make things easier. The number of API queries one has to make in order to get data […], the learning curve about SPARQL and RDF (although, you can ignore both, unless you want to use them explicitly - you can just use JSON and the Wikidata API), the opaqueness of the identifiers (wdt:P25 wd:Q9682 instead of “mother” and “Queen Elizabeth II”) were just a few. The documentation seems hard to find, there seem to be a lack of libraries and APIs that are easy to use. And yet, comments like "if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you" surprised me a lot. […] I am not here to fight. I am here to listen and to learn, in order to help figuring out what needs to be made better. Again, very commendable! Almost an opening to really understanding the perspective of casual potential users. But then: the entire rest of the post does not really address "the other side", and instead completely focuses on the kinds of things Wikidata enthusiasts care about: comparing Wikipedia and Wikidata quality in this example, etc. I mean, sure, this query he presents is short: select * { wd:Q9682 (wdt:P25|wdt:P22)* ?p . ?p wdt:P25|wdt:P22 ?q } but when he says: > I would claim that I invested far less work than Bill in creating my graph data. No data cleansing, no scraping, no crawling, no entity reconciliation, no manual checking. he's ignoring the work he invested in learning that query language (and where to query it), for instance. And this post would have been a perfect opportunity to teach readers about how to go from the question "all ancestors of Queen Elizabeth" to that query (and in trying to teach it, he may have better discovered exactly what is hard about it), but he just squanders the opportunity (just as when he says "plenty of libraries" without inviting exploration by linking to the easiest one): this is a typical thing enthusiasts do, which is unfortunate IMO. When scraping HTML from Wikipedia, one is using general-purpose well-known tools. You'll get slightly better at whatever general-purpose programming language and libraries you were using, learn something that may be useful the next time you need to scrape something else. And most importantly, you know that you'll finish, you can see a path to success. When exploring something "alternative" like Wikidata, you aren't sure if it will work, so the alternative path needs to work harder to convince potential users of success. --- Personal story: I actually know about the existence of Wikidata. Yet the one time I tried to use it, I couldn't figure out how. This is what I was trying to do: plot a graph of the average age of Turing Award winners by year. (Reproduce the first figure from here: http://hagiograffiti.blogspot.com/2009/01/when-will-singularity-occur.html http://hagiograffiti.blogspot.com/2009/01/when-will-singular... just for fun) One would think this is a perfect use-case for Wikidata: presumably it has a way of going from Turing Award → list of winners → each winner's date of birth. But I was stymied at the very first step: despite knowing of the existence of Wikidata, and being able to go from the Wikipedia page that lists all recipients (current version: https://en.wikipedia.org/w/index.php?title=Turing_Award&oldid=1028806461#Recipients https://en.wikipedia.org/w/index.php?title=Turing_Award&oldi... ) to the Wikidata item for "Turing Award" (look for "Wikidata item" in the sidebar on the left) https://www.wikidata.org/wiki/Q185667 https://www.wikidata.org/wiki/Q185667 I could not quickly find a way of getting a list of recipients from there. Tantalizingly, the data maybe does exist e.g. if I go to one of the recipients like Leslie Valiant https://www.wikidata.org/wiki/Q93154 https://www.wikidata.org/wiki/Q93154 I see a "statement" award received → Turing Award with "property" point in time → 2010. Even after coming so close, and being interested in using Wikidata, it was not easy enough for me to get to the next step (which I still imagine is possible, maybe with tens of minutes of effort), until I just decided "screw this, I'll just scrape the Wikipedia page" (I scraped the wikisource rather than html). And if one is going to have to scrape anyway, then might as well do the rest too (dates of birth) with scraping.

nine_k 5y ago

I think this project is not getting enough attention: https://github.com/zverok/wikipedia_ql https://github.com/zverok/wikipedia_ql It allows to query Wikipedia (not wikidata, but the actual human-readable text) more or less directly, mixing the way you describe a scraper with some nicer higher-level constructs. Can't vouch for its performance, but the API is interesting and nice.

aasasd 5y ago
My personal long-standing wish is querying categories, in which the pages have the same infoboxes, by the fields in the boxes. Preferably without waiting to download dozens or hundreds of pages first. The infobox→Wikidata integration would pretty much solve that (not the other way around), and I'm told that the Wikidata Bridge project aims to do that integration: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge However, if someone made a different database that would be queryable tomorrow, I wouldn't mind.
- valexiev 5y ago
  Use https://www.mediawiki.org/wiki/Wikidata_Query_Service/Categories https://www.mediawiki.org/wiki/Wikidata_Query_Service/Catego... by making a federated query to https://query.wikidata.org/bigdata/namespace/categories/sparql https://query.wikidata.org/bigdata/namespace/categories/spar.... Cheers!
  - aasasd 5y ago
    Thanks, but this sounds like it only says what subcategories and, perhaps, pages are in the categories—but doesn't contain any data from the pages themselves. My main target is kinda-structured data from infoboxes—e.g. genre, platform, year for videogames. I don't even need categories particularly—I just grab all pages from them, hoping that all the pages I would want are in these categories.
- vrandecic 5y ago
  Sounds a lot like DBpedia to me

ComputerGuru 5y ago

Site is down. Archive.org link: https://web.archive.org/web/20210821004307/http://simia.net/wiki/Wikidata_or_scraping_Wikipedia https://web.archive.org/web/20210821004307/http://simia.net/...

simonw 5y ago

Here-in lies the problem for me: select * { wd:Q9682 (wdt:P25|wdt:P22)* ?p . ?p wdt:P25|wdt:P22 ?q } - I am extremely motivated to learn how to use this: I have a deep desire to extract data from Wikipedia, and I'm fascinated by graph databases. And yet, despite trying on several previous occasions, SPARQL has completely failed to stick in my brain. This is partly my own failing: I'm confident that if I really dedicated myself to it I could get over this hump. But it's also a sign that the learning curve here really is tremendously steep, which I think indicates a problem with the design of the technology.

mmarx 5y ago
A few days ago, the Wikidata Query Builder[0] was deployed. It provides a visual interface to generate simple SPARQL queries, and you can show the generated queries. Maybe this can help you in understanding how SPARQL patterns work? [0] https://query.wikidata.org/querybuilder/ https://query.wikidata.org/querybuilder/
- simonw 5y ago
  That does look like a big step forward. It could really benefit from some linked examples on that page though - I stared at the interface for quite a while, unable to figure out how to use it for anything - then I dug around for an example link and it started to make sense to me. https://query.wikidata.org/querybuilder/?query=%7B%22conditions%22%3A%5B%7B%22propertyId%22%3A%22P1435%22%2C%22propertyDataType%22%3A%22wikibase-item%22%2C%22propertyValueRelation%22%3A%22matching%22%2C%22referenceRelation%22%3A%22regardless%22%2C%22value%22%3A%22Q10729054%22%2C%22subclasses%22%3Afalse%2C%22conditionRelation%22%3Anull%2C%22negate%22%3Afalse%7D%2C%7B%22propertyId%22%3A%22P131%22%2C%22propertyDataType%22%3A%22wikibase-item%22%2C%22propertyValueRelation%22%3A%22regardless-of-value%22%2C%22referenceRelation%22%3A%22regardless%22%2C%22value%22%3A%22%22%2C%22subclasses%22%3Atrue%2C%22conditionRelation%22%3A%22and%22%2C%22negate%22%3Afalse%7D%5D%2C%22limit%22%3A100%2C%22useLimit%22%3Atrue%2C%22omitLabels%22%3Afalse%7D https://query.wikidata.org/querybuilder/?query=%7B%22conditi...
dmitriid 5y ago
Or use something like https://github.com/zverok/wikipedia_ql https://github.com/zverok/wikipedia_ql that uses Mediawiki API

nathell 5y ago

Wikipedia asks people not to crawl it. There are database dumps that you can instead import into your local MySQL and work from there. https://en.wikipedia.org/wiki/Wikipedia:Database_download#Please_do_not_use_a_web_crawler https://en.wikipedia.org/wiki/Wikipedia:Database_download#Pl...

bawolff 5y ago
Wikipedia has no objection to crawling a couple thousand pages if you do so at a reasonable speed and set a user-agent with a contact email. If you want to crawl millions of pages please use a dump.

elcapitan 5y ago

How do other people use Wikidata dumps if they are not using the "official" (with sparql or so) way of querying it? I have done some pretty raw extraction from it (e.g. download the already pretty large zipped json dump, then unzip it on the fly and parse the json, and extract triples and entities). Not sure if that is really quite efficient, but the dumps are hard to work with, and I really just needed the entities in one language and the triples/graph of them.

brokensegue 5y ago
They release a json dump but also just a dump of true triples in nt format (basically just tsv). It's few tens of gigs uncompressed
IAmEveryone 5y ago
This tool https://wdumps.toolforge.org/ https://wdumps.toolforge.org/ allows you to create bespoke dumps that are pre-filtered for your needs.

drej 5y ago

It’s fairly difficult from the other side as well - contributing. I’ve been trying to complete wikidata from a few open source datasets I am intensly familiar with… and it’s been rather painful. WD is the sole place I have ever interacted with that uses RDF, so I always forget the little syntax I learned last time around. I have some pre-existing queries versioned, because I’ll never be able to write them again. I even went to a local Wikimedia training to get acquainted with some necessary tooling, but I’m still super unproductive compared to e.g. SQL. It’s sad, really, I’d love to contribute more, but the whole data model is so clunky to work with.

valexiev 5y ago
I have no problems with the data model, but sadly you can't insert RDF statements: you have to go through tools like QS and wikidata-cli and the WD update performance is dismal. See https://phabricator.wikimedia.org/T290061 https://phabricator.wikimedia.org/T290061, which I posted in https://phabricator.wikimedia.org/project/board/5504/ https://phabricator.wikimedia.org/project/board/5504/ for DataQualityDays2021
drej 5y ago
That being said, I now remember I stopped contributing for a slightly different reason. While I tried to fill WD with complete information about a given subject, this was never leverage by a Wikimedia project - there is certain resistance to generating Wikipedia articles/infoboxes from Wikidata, so you're fighting on two fronts, you always have to edit things in two places and it's just a waste of everyone's time. Unless the attitude becomes "all facts in infoboxes and most tables come from WD", the two "datasets" will continue diverging. That is obviously more easily said than done, because relying on WD makes Wikipedia contribution a lot more difficult... and that pretty much defeats its purpose.
- dane-pgp 5y ago
  > the two "datasets" will continue diverging. You may be pleased to learn that there is a project underway that aims to largely solve that problem: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge The last piece of news I can immediately find is that it was deployed to the Catalan Wikipedia in August 2020, but I'm not sure what progress there has been since.

dheera 5y ago

I'd highly recommend parsing the mobile edition of Wikipedia, it is much easier to parse.

marto1 5y ago

For me, I ended up using both. Opting for wikidata wherever made sense, but a lot of things felt half built/broken.

bob229 5y ago

Wikipedia is garbage.

aasasd 5y ago

I'm all for Wikidata, it's great in some ‘high-profile’ cases like data on countries—at least by my moderate standards. I didn't have much problem with Sparql, or perhaps my queries were simple. However, once you get into the lowbrow territory of e.g. modern cultural artifacts, people just edit Wikipedia way more, end of story. Want to know what games of some genre were made for platforms of your choice, sorted by year? You go to Wikipedia, not Wikidata. I'm told that there's a project to integrate infoboxes with Wikidata, so that their info goes into Wikidata when edited (not the other way around)—which would solve a large part of this scarcity, if the integration is seamless enough. Haven't yet seen it in action. Here's the project, Wikidata Bridge: https://www.mediawiki.org/wiki/Wikidata_Bridge https://www.mediawiki.org/wiki/Wikidata_Bridge

aaron695 5y ago

Wikidata has failed. This is not their fault, we have know for 10+ years machine readable data fails. See tags, and their failure (hashtag != tag) Their example of a goat, a simple common animal, does not identify how many teats a goat has ( does it? ) https://www.wikidata.org/wiki/Q2934 https://www.wikidata.org/wiki/Q2934 https://en.wikipedia.org/wiki/Goat https://en.wikipedia.org/wiki/Goat Everyone starting does family trees. Because they are easy and easily defined. But even this articles query ends up at fictional characters. The "period of lactation" for a goat is not a number. And it's not even one graph. It's multiple graphs which we don't have the data to accurately know. The original article was 100% correct. Web-scraping was the way to get the data. Web-scraping is a very useful and transferable skill. There's no point learning skills on a known failed idea like machine readable data. Wikipedia allows web scraping, anyone who tells you different is lying, see their robots.txt to make sure you don't get rate limited if doing massive amounts https://en.wikipedia.org/robots.txt https://en.wikipedia.org/robots.txt (and to see the stuff they don't want you to read). They also have downloadable dumps you can use.

shp0ngle 5y ago

for me, as an occasional wikipedia editor, wikidata is just the annoying thing I need to deal with when linking different language wikis together. I can never figure out anything there, but it’s just the UX I guess

denton-scratch 5y ago

My attention was caught by "Albert The Bear" (second list). It reminded me of Albert And The Lion: https://www.youtube.com/watch?v=oaw-savyK0s https://www.youtube.com/watch?v=oaw-savyK0s