7 ms·

Data-Mining Wikipedia for Fun and Profit

207 points by billpg 5y ago

deleted 5y ago
[deleted]
papercrane 5y ago
For this particular problem I wonder if wikidata would be better instead of scraping the HTML.
- billpg 5y ago
  (Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.
  - mistrial9 5y ago
    earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over
    - udp 5y ago
      The ivory tower is working on it: https://github.com/w3c/EasierRDF https://github.com/w3c/EasierRDF
      - tasogare 5y ago
        RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
        namedgraph 5y ago
        Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.
        tasogare 5y ago
        Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format. And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.
        namedgraph 5y ago
        Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not. You're right about bioinformatics, but lets do a quick check on http://sparql.club/ http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.
        udp 5y ago
        I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated. > Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.
      - liminal 5y ago
        I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.
    - bawolff 5y ago
      Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that. In real life you use tools for both.
      - namedgraph 5y ago
        Triples are already structured, machine-readable data. HTML is not.
  - shock-value 5y ago
    Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 https://www.wikidata.org/wiki/Q16156475).
    - boxed 5y ago
      I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/ https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell. I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked. The code to do that is at https://github.com/boxed/relatedhow https://github.com/boxed/relatedhow
      - shock-value 5y ago
        There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever https://github.com/ad-freiburg/QLever) Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.
  - 3pt14159 5y ago
    Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew. When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.
    - tcgv 5y ago
      > Don't worry about the haters To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite. But I know it's pretty common to see haters nitpicking things all around ;)
  - papercrane 5y ago
    Hard to use it if you don't know about it! I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.
  - hutzlibu 5y ago
    Yup, I had the same situation some months before, even though I knew wikidata was a thing. I know javascript and had the pages at hand. I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.
- powera 5y ago
  Definitely. There's even a public query service ( https://query.wikidata.org/ https://query.wikidata.org/ ) which can do a lot of this (though SQL is not good with searching for chains).
  - deleted 5y ago
    [deleted]
  - bawolff 5y ago
    Note: that's the sparql endpoint. Much better at searching chains than SQL (but easy to make slow queries that timeout). The SQL endpoint is at https://quarry.wmflabs.org/ https://quarry.wmflabs.org/ however it doesn't have the actual data so much as metadata (mostly) so its not super useful.
- brodo 5y ago
  It’s so sad that almost nobody knows or uses SPARQL…
  - devbas 5y ago
    Because the syntax is relatively complex and it is difficult to judge which endpoints and definitions to use.
    - ZeroGravitas 5y ago
      I learned SPARQL recently, and would agrre its complicated to get info out of Wikidata. However, having read the article, they didnt have an easy time with scraping Wikipedia either. So I'd probably still recommend people look into wikidata and SPARQL if they want to do this kind of thing. Theres a few tools that generate queries for you, and some cli tools as well: https://github.com/maxlath/wikibase-cli#readme https://github.com/maxlath/wikibase-cli#readme It makes Wikipedia better too, in a virtuous cycle, with some infoboxes like those that he scraped being converted to be automatically populated from wikidata.
  - matkoniecz 5y ago
    In my experience SPARQL is really hard to use, and Wikidata data quality is really low. To the point that one of my larger project is trying to filter data to make it usable for my usecase. Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_Konieczny https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ). But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it. Queries as simple as "is this entry describing event, bridge or neither" are requiring extreme effort to get right in a reliable way, including maintaining private list of patches and exemptions. And bots creating millions of known duplicated entries and expecting people to resolve this manually is quite discouraging. Creating Wikidata entries for Cebuano Wikipedia 'articles' was accepted, despite that Cebuano botpedia is nearly completely bot-generated. And that is without unclear legal status. Yes, they can legally import databases covered by database rights - but they should either make clear that Wikidata is a legal quagmire in EU or forbid such imports. But Wikidata community did neither.
- squeaky-clean 5y ago
  Like people on the other comment have said, if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you.
- dalf 5y ago
  Yes: * http://www.entitree.com/en/family_tree/Elizabeth_II http://www.entitree.com/en/family_tree/Elizabeth_II * https://family.toolforge.org/ancestors.php?q=Q187114 https://family.toolforge.org/ancestors.php?q=Q187114 Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/en https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/... --- Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/queries/examples https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... --- Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).
  - zaik 5y ago
    There is a tool to get infobox data from Wikipedia into Wikidata: https://pltools.toolforge.org/harvesttemplates/ https://pltools.toolforge.org/harvesttemplates/ The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).
    - beardyw 5y ago
      So am I right in thinking Wikidata is sort of scraped from Wikipedia internally? The other way round seems better, but obviously too late.
      - dalf 5y ago
        A lot of items don't have a wikipedia page, but all wikipedia, wikitravel, wiki* pages have a wikidata item (see "Wikidata item" link of the left side of the pages). Some wikipedia infoxes are based on wikidata. I can't find an example, but here some links: * https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_help https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_... * https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox * https://en.wikipedia.org/wiki/Template:Infobox_person/Wikidata https://en.wikipedia.org/wiki/Template:Infobox_person/Wikida... There are lexeme too, and it is not based on wiktionary. Search the prefix "L:" (without quote). Example: * https://www.wikidata.org/w/index.php?search=L%3Acat&search=L%3Acat&title=Special%3ASearch&go=Go&ns0=1&ns120=1 https://www.wikidata.org/w/index.php?search=L%3Acat&search=L... * https://www.wikidata.org/wiki/Lexeme:L7 https://www.wikidata.org/wiki/Lexeme:L7 Also, there are a lot of tools on toolforge.org. One is reasonator which produce sentences from a wikidata item: https://reasonator.toolforge.org/?q=Q1339 https://reasonator.toolforge.org/?q=Q1339
- mark_l_watson 5y ago
  +1 use WikiData’s SPARQL endpoint. Still, it was a cool article and a good example of scraping information.
humanistbot 5y ago
Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps (https://dumps.wikimedia.org https://dumps.wikimedia.org) to the fully-featured API with a nice sandbox (https://en.wikipedia.org/wiki/Special:ApiSandbox https://en.wikipedia.org/wiki/Special:ApiSandbox) and Wikidata (https://wikidata.org https://wikidata.org).
- whall6 5y ago
  Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
  - michaelbuckbee 5y ago
    That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
    - billpg 5y ago
      (Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
      - SlimyHog 5y ago
        The HTML is more volatile and subject to change than other sources though
        FalconSensei 5y ago
        Don't remember the last time wikipedia changed the infobox though
    - jcun4128 5y ago
mrVentures 5y ago
Lol I love that it was edited out right away. I probably agree with that decision but this was a really cool project. I have followed a few YouTube channels which make graphic visualizations for Generations and I think they would really appreciate if you shared this Tech with them
lizhang 5y ago
Where's the profit?
- ad404b8a372f2b9 5y ago
  Good question, as a data-hoarder mining has always been fun but I was interested in seeing the profit. I've yet to find any monetary value in the troves of data I accumulate.
- billpg 5y ago
  For three minutes, something I wrote was cited on Wikipedia.
  - k12sosse 5y ago
    Ah yes, every student in my district had this distinction too, until they lifetime banned our egress IPs over it. The person who did the revert seems like a bundle of joy, though. Congrats on your efforts.
billpg 5y ago
(Author here) In an act of divine justice, my website is down. https://web.archive.org/web/20210711201037/https://billpg.com/data-mining-wikipedia/ https://web.archive.org/web/20210711201037/https://billpg.co... (I'll send you a donation. Thank you!)
barbinbrad 5y ago
For anyone who is interested: I used to work with a guy named Richard Wang, who indexed Wikipedia as his training data set in order to do named entity recognition. He'd be a good person to talk for anyone pursuing this. Here's a demo: https://www.youtube.com/watch?v=SyhaxCjrZFw https://www.youtube.com/watch?v=SyhaxCjrZFw
ahP7Deew 5y ago
Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...
- kemayo 5y ago
  Parsoid[1] is what you'd want for that, most likely. It's the new wikitext parser that MediaWiki is gradually switching over to, but it has the virtue of being usable entirely outside of MediaWiki if you need to. [1]: https://www.mediawiki.org/wiki/Parsoid https://www.mediawiki.org/wiki/Parsoid
  - bawolff 5y ago
    Yes, but it implements lua by calling into the old stack, which doesn't really solve OPs problem.
    - kemayo 5y ago
      Okay, yeah, I guess that Lua modules in particular are going to make everything fall down.
ygmelnikova 5y ago
If the task is to answer the question "Does a royal lineage exist between Alfred the Great and Queen Elizabeth?" then this works fine, but the results only show 1 of possibly 100s of such paths. If you're of European descent, it's 100% possible for you to find a similar connection to Alfred the Great as well :)
permo-w 5y ago
>The graph was interesting but this wasn’t the primary objective of this exercise. I wanted to write “He is the n-times great-father of his current successor Queen Elizabeth.” on King Alfred’s Wikipedia page. Wouldn’t that contravene Wikipedia’s rules on original research?
- bawolff 5y ago
  I also think its of questionable relavence to the topic at hand. Just because a fact about something is true doesn't mean it should be in the article on the topic. But yes, citing your own blog isn't a valid source, and neither is citing wikipedia - https://en.wikipedia.org/wiki/Wikipedia:Verifiability#Self-published_sources https://en.wikipedia.org/wiki/Wikipedia:Verifiability#Self-p...
- Mathnerd314 5y ago
  It's not original research, more like synthesis. Reading the policy he's probably OK under the routine calculation exception. The edit got reverted because there was no clear criteria for mentioning Queen Elizabeth III but not the other descendants. If he made an info graphic or info box with a clear inclusion criteria and stuck it in the Alfred article it would probably stick.
deleted 5y ago
[deleted]
beardyw 5y ago
I have a basically English heritage. If I counted unrelated ancestors back through 45 generations I reckon I should have had 35,184,372,088,832! Since the population of England back in the 800s was probably a few million, what are the chances I am NOT a descendent of King Alfred?
vrandecic 5y ago
I found that work very interesting. So I downloaded the data by OP (thanks for publishing it!) and compared it with Wikidata. The results are here: http://simia.net/wiki/Main_Page http://simia.net/wiki/Main_Page
vagab0nd 5y ago
Loosely related: I had some fun working on a database with all events parsed from Wikipedia [0]. I parsed the entire wiki texts using a language model, extracting date/time/location and what happened then and there. The location data was then fed into a local openstreetmap instance to get the coordinates. I then built a front end to allow querying events near by, in time or in space. The UI is a bit clunky because my database is too large and I had little experience working with them. But building it was fun. 0: https://whataday.info/ https://whataday.info/

207 points by billpg 5y ago

deleted 5y ago

[deleted]

papercrane 5y ago

For this particular problem I wonder if wikidata would be better instead of scraping the HTML.

billpg 5y ago
(Author here.) Perhaps, but I already know how to scrape HTML and I know the data I wanted to pull out was in there. I have no idea how to query wikidata and it could have ended up being a blind alley. Also, it was only my reading your comment just now that told me wikidata was even a thing.
- mistrial9 5y ago
  earth calling ivory tower, earth calling ivory tower superior RDF triples are like martian language to millions of humans over
  - udp 5y ago
    The ivory tower is working on it: https://github.com/w3c/EasierRDF https://github.com/w3c/EasierRDF
    - tasogare 5y ago
      RDF has to be the best and saddest example of sunk cost fallacy. Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers, its cultists are double downing on their abstruse technology stack, making it always more complicated while still not addressing any of its fundamental problems.
      - namedgraph 5y ago
        Let us see your proposal of the superior model? RDF was designed primarily for data interchange and there's nothing that beats it at that.
        tasogare 5y ago
        Nothing beats it for data exchange? You must be joking, because if it were remotely true RDF would be in wide use, which is totally not. Except a few niche domains like bioinformatics, it is not used. No killer application use it as a data format, no popular data format is based on it either. Actually I can think of a single data format based on RDF, and the only open-data I know which use it have been converted to it and was simplier to use in their original format. And for the model: property graph. But yeah, enjoy your Stockholm syndrome with your model where reification is required to annotate an edge. Also even your nickname is an aknowledgment of RDF failure: named graphs (n-quads) were created because RDF triples aren't good enough for modeling data.
        namedgraph 5y ago
        Yes, let us see how you do data interchange without global identifiers. Such as URIs, which RDF has built-in natively and property graphs do not. You're right about bioinformatics, but lets do a quick check on http://sparql.club/ http://sparql.club/ on who else is looking for RDF/SPARQL specialists. Oh look: automotive industry, finance, publishing, medical, research etc.
      - udp 5y ago
        I mean, imho RDF isn't the problem. RDF itself is very simple. As you correctly point out, the stack is overcomplicated. > Instead of redirecting their efforts to a more general graph model which has actual hype and use by developers neo4j is basically this. You can also load RDF into neo4j using neosemantics and query it using Cypher instead of using a conventional triplestore with SPARQL, which is nice.
    - liminal 5y ago
      I just looked at the linked repo. Have they made any progress? It looks like _very_ early days.
  - bawolff 5y ago
    Are you saying parsing html is easier than parsing rdf triples? Because i dont know about that. In real life you use tools for both.
    - namedgraph 5y ago
      Triples are already structured, machine-readable data. HTML is not.
- shock-value 5y ago
  Generally Wikidata would definitely be the way to go here, though I just now tried to retrace your graph in Wikidata and it seems to be missing at least one relation (Ada of Holland has no children listed -- https://www.wikidata.org/wiki/Q16156475 https://www.wikidata.org/wiki/Q16156475).
  - boxed 5y ago
    I am doubtful. I tried for a long time to use it to get data or for my taxonomic graph project (https://relatedhow.kodare.com/ https://relatedhow.kodare.com/) and SPARCQL was just not usable at all. The biggest problem was the 60s time limit. Totally not workable for what I wanted. I also had issues with seemingly inconsistent results, but it was hard to tell. I ended up loading the full nightly db dump and filtering it streaming from the zip instead. Faster and it actually worked. The code to do that is at https://github.com/boxed/relatedhow https://github.com/boxed/relatedhow
    - shock-value 5y ago
      There's an alternate Wikidata query engine available here: https://qlever.cs.uni-freiburg.de/wikidata https://qlever.cs.uni-freiburg.de/wikidata (from https://github.com/ad-freiburg/QLever https://github.com/ad-freiburg/QLever) Currently it doesn't support some SPARQL features, but I've found it to generally be quite a bit faster for most queries.
- 3pt14159 5y ago
  Don't worry about the haters. You needed a paltry amount of data and you got it with the tools you had and knew. When I was analyzing Wikipedia about 10 years ago for fun and, later, actual profit. I did the responsible thing and downloaded one of their megadumps because I needed every English page. That's what people here are concerned about, but it doesn't matter for your use case.
  - tcgv 5y ago
    > Don't worry about the haters To be fair, the original comment just made a valid observation in a casual way, he didn't criticize the approach of the OP, nor was he impolite. But I know it's pretty common to see haters nitpicking things all around ;)
- papercrane 5y ago
  Hard to use it if you don't know about it! I only thought of it myself because you mentioned the problem with deducing which parent is the mother and which is the father, and I remember in wikidata those are separate fields.
- hutzlibu 5y ago
  Yup, I had the same situation some months before, even though I knew wikidata was a thing. I know javascript and had the pages at hand. I looked at wikidata and some pages about, but still had no clear idea how to use it and no motivation to digg into it. Because js just worked with a small custom script, to retrieve some pages and data.
powera 5y ago
Definitely. There's even a public query service ( https://query.wikidata.org/ https://query.wikidata.org/ ) which can do a lot of this (though SQL is not good with searching for chains).
- deleted 5y ago
  [deleted]
- bawolff 5y ago
  Note: that's the sparql endpoint. Much better at searching chains than SQL (but easy to make slow queries that timeout). The SQL endpoint is at https://quarry.wmflabs.org/ https://quarry.wmflabs.org/ however it doesn't have the actual data so much as metadata (mostly) so its not super useful.
brodo 5y ago
It’s so sad that almost nobody knows or uses SPARQL…
- devbas 5y ago
  Because the syntax is relatively complex and it is difficult to judge which endpoints and definitions to use.
  - ZeroGravitas 5y ago
    I learned SPARQL recently, and would agrre its complicated to get info out of Wikidata. However, having read the article, they didnt have an easy time with scraping Wikipedia either. So I'd probably still recommend people look into wikidata and SPARQL if they want to do this kind of thing. Theres a few tools that generate queries for you, and some cli tools as well: https://github.com/maxlath/wikibase-cli#readme https://github.com/maxlath/wikibase-cli#readme It makes Wikipedia better too, in a virtuous cycle, with some infoboxes like those that he scraped being converted to be automatically populated from wikidata.
- matkoniecz 5y ago
  In my experience SPARQL is really hard to use, and Wikidata data quality is really low. To the point that one of my larger project is trying to filter data to make it usable for my usecase. Yes, I made some improvements ( https://www.wikidata.org/wiki/Special:Contributions/Mateusz_Konieczny https://www.wikidata.org/wiki/Special:Contributions/Mateusz_... ). But overall I would not encourage using it, if I would know how much work it takes to get usable data I would not bother with it. Queries as simple as "is this entry describing event, bridge or neither" are requiring extreme effort to get right in a reliable way, including maintaining private list of patches and exemptions. And bots creating millions of known duplicated entries and expecting people to resolve this manually is quite discouraging. Creating Wikidata entries for Cebuano Wikipedia 'articles' was accepted, despite that Cebuano botpedia is nearly completely bot-generated. And that is without unclear legal status. Yes, they can legally import databases covered by database rights - but they should either make clear that Wikidata is a legal quagmire in EU or forbid such imports. But Wikidata community did neither.
squeaky-clean 5y ago
Like people on the other comment have said, if you've actually tried getting data from wikidata/wikipedia you very quickly learn the HTML is much easier to parse than the results wikidata gives you.
dalf 5y ago
Yes: * http://www.entitree.com/en/family_tree/Elizabeth_II http://www.entitree.com/en/family_tree/Elizabeth_II * https://family.toolforge.org/ancestors.php?q=Q187114 https://family.toolforge.org/ancestors.php?q=Q187114 Tools found on this page: https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/en https://www.wikidata.org/wiki/Wikidata:Tools/Visualize_data/... --- Some SPARQL queries: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/queries/examples https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... --- Out of topic: I wish wikipedia would provide an API to get the infoboxes (made using Lua or wikidata).
- zaik 5y ago
  There is a tool to get infobox data from Wikipedia into Wikidata: https://pltools.toolforge.org/harvesttemplates/ https://pltools.toolforge.org/harvesttemplates/ The easily parsable Infobox data can probably already be found in Wikidata (assuming there is a property).
  - beardyw 5y ago
    So am I right in thinking Wikidata is sort of scraped from Wikipedia internally? The other way round seems better, but obviously too late.
    - dalf 5y ago
      A lot of items don't have a wikipedia page, but all wikipedia, wikitravel, wiki* pages have a wikidata item (see "Wikidata item" link of the left side of the pages). Some wikipedia infoxes are based on wikidata. I can't find an example, but here some links: * https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_help https://commons.wikimedia.org/wiki/Commons:Wikidata_infobox_... * https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox https://commons.wikimedia.org/wiki/Template:Wikidata_Infobox * https://en.wikipedia.org/wiki/Template:Infobox_person/Wikidata https://en.wikipedia.org/wiki/Template:Infobox_person/Wikida... There are lexeme too, and it is not based on wiktionary. Search the prefix "L:" (without quote). Example: * https://www.wikidata.org/w/index.php?search=L%3Acat&search=L%3Acat&title=Special%3ASearch&go=Go&ns0=1&ns120=1 https://www.wikidata.org/w/index.php?search=L%3Acat&search=L... * https://www.wikidata.org/wiki/Lexeme:L7 https://www.wikidata.org/wiki/Lexeme:L7 Also, there are a lot of tools on toolforge.org. One is reasonator which produce sentences from a wikidata item: https://reasonator.toolforge.org/?q=Q1339 https://reasonator.toolforge.org/?q=Q1339
mark_l_watson 5y ago
+1 use WikiData’s SPARQL endpoint. Still, it was a cool article and a good example of scraping information.

humanistbot 5y ago

Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps (https://dumps.wikimedia.org https://dumps.wikimedia.org) to the fully-featured API with a nice sandbox (https://en.wikipedia.org/wiki/Special:ApiSandbox https://en.wikipedia.org/wiki/Special:ApiSandbox) and Wikidata (https://wikidata.org https://wikidata.org).

whall6 5y ago
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
- michaelbuckbee 5y ago
  That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
  - billpg 5y ago
    (Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
    - SlimyHog 5y ago
      The HTML is more volatile and subject to change than other sources though
      - FalconSensei 5y ago
        Don't remember the last time wikipedia changed the infobox though
  - jcun4128 5y ago

mrVentures 5y ago

Lol I love that it was edited out right away. I probably agree with that decision but this was a really cool project. I have followed a few YouTube channels which make graphic visualizations for Generations and I think they would really appreciate if you shared this Tech with them

lizhang 5y ago

Where's the profit?

ad404b8a372f2b9 5y ago
Good question, as a data-hoarder mining has always been fun but I was interested in seeing the profit. I've yet to find any monetary value in the troves of data I accumulate.
billpg 5y ago
For three minutes, something I wrote was cited on Wikipedia.
- k12sosse 5y ago
  Ah yes, every student in my district had this distinction too, until they lifetime banned our egress IPs over it. The person who did the revert seems like a bundle of joy, though. Congrats on your efforts.

billpg 5y ago

(Author here) In an act of divine justice, my website is down. https://web.archive.org/web/20210711201037/https://billpg.com/data-mining-wikipedia/ https://web.archive.org/web/20210711201037/https://billpg.co... (I'll send you a donation. Thank you!)

barbinbrad 5y ago

For anyone who is interested: I used to work with a guy named Richard Wang, who indexed Wikipedia as his training data set in order to do named entity recognition. He'd be a good person to talk for anyone pursuing this. Here's a demo: https://www.youtube.com/watch?v=SyhaxCjrZFw https://www.youtube.com/watch?v=SyhaxCjrZFw

ahP7Deew 5y ago

Has anyone found an easy way to expand their templates without using their whole stack? I tried getting Lua templates working from Python but didn't get very far...

kemayo 5y ago
Parsoid[1] is what you'd want for that, most likely. It's the new wikitext parser that MediaWiki is gradually switching over to, but it has the virtue of being usable entirely outside of MediaWiki if you need to. [1]: https://www.mediawiki.org/wiki/Parsoid https://www.mediawiki.org/wiki/Parsoid
- bawolff 5y ago
  Yes, but it implements lua by calling into the old stack, which doesn't really solve OPs problem.
  - kemayo 5y ago
    Okay, yeah, I guess that Lua modules in particular are going to make everything fall down.

ygmelnikova 5y ago

If the task is to answer the question "Does a royal lineage exist between Alfred the Great and Queen Elizabeth?" then this works fine, but the results only show 1 of possibly 100s of such paths. If you're of European descent, it's 100% possible for you to find a similar connection to Alfred the Great as well :)

permo-w 5y ago

>The graph was interesting but this wasn’t the primary objective of this exercise. I wanted to write “He is the n-times great-father of his current successor Queen Elizabeth.” on King Alfred’s Wikipedia page. Wouldn’t that contravene Wikipedia’s rules on original research?

bawolff 5y ago
I also think its of questionable relavence to the topic at hand. Just because a fact about something is true doesn't mean it should be in the article on the topic. But yes, citing your own blog isn't a valid source, and neither is citing wikipedia - https://en.wikipedia.org/wiki/Wikipedia:Verifiability#Self-published_sources https://en.wikipedia.org/wiki/Wikipedia:Verifiability#Self-p...
Mathnerd314 5y ago
It's not original research, more like synthesis. Reading the policy he's probably OK under the routine calculation exception. The edit got reverted because there was no clear criteria for mentioning Queen Elizabeth III but not the other descendants. If he made an info graphic or info box with a clear inclusion criteria and stuck it in the Alfred article it would probably stick.

deleted 5y ago

[deleted]

beardyw 5y ago

I have a basically English heritage. If I counted unrelated ancestors back through 45 generations I reckon I should have had 35,184,372,088,832! Since the population of England back in the 800s was probably a few million, what are the chances I am NOT a descendent of King Alfred?

vrandecic 5y ago

I found that work very interesting. So I downloaded the data by OP (thanks for publishing it!) and compared it with Wikidata. The results are here: http://simia.net/wiki/Main_Page http://simia.net/wiki/Main_Page

vagab0nd 5y ago

Loosely related: I had some fun working on a database with all events parsed from Wikipedia [0]. I parsed the entire wiki texts using a language model, extracting date/time/location and what happened then and there. The location data was then fed into a local openstreetmap instance to get the coordinates. I then built a front end to allow querying events near by, in time or in space. The UI is a bit clunky because my database is too large and I had little experience working with them. But building it was fun. 0: https://whataday.info/ https://whataday.info/