6 ms·

Json-wikipedia

40 points by ikuyamada 13y ago

andrewvc 13y ago
Some might be interested in https://github.com/andrewvc/wikiparse https://github.com/andrewvc/wikiparse , code I wrote to dump wikipedia into elasticsearch. Here's a chapter in my book on searching it: http://exploringelasticsearch.com/book/searching-natural-language/searching-the-wikipedia-dataset.html http://exploringelasticsearch.com/book/searching-natural-lan...
sktrdie 13y ago
What about DBPedia http://dbpedia.org/page/The_Shining_(film) http://dbpedia.org/page/The_Shining_(film) ? That's available as JSON, and much larger community/effort behind it.
- brianshaler 13y ago
  There's a much larger community/effort behind DBPedia than Wikipedia? Or are you comparing DBPedia to this utility that converts Wikipedia data dumps? Edit: looks like I need to do some more research into DBPedia. It sounds like that community has created a dataset that categorizes Wikipedia article (e.g. "things" and "facts")
- exg 13y ago
  Correct me if I am wrong, but I do not think that DBPedia includes the full Wikipedia articles (only extended abstracts). So I believe that json dumps of DBPedia and Wikipedia dumps produced using this tool are two different things.
- ikuyamada 13y ago
  In my understanding, DBpedia is a project for extracting data mainly from Wikipedia infoboxes (not whole data dumps) by collaboratively creating rules for converting data into cleaner schema that enables to perform a query such as SPARQL. I think this is a project that directly converts wikipedia dump xml to JSON for easier manipulation, which differs from DBpedia.
- diegolo 13y ago
  Hi, I'm the author of Json-Wikipedia. It's different from DB-Pedia. DBPedia converts the semistructured data containined in the infoboxes in rdf triples, while Json-Wikipedia converts ALL the content of the page (text, infoboxes, links) into a JSON object. I did this because I need some features that were not available on dbpedia (e.g., the whole text of an article, all the anchors referring to an article etc etc). Also Json-Wikipedia simplifies the processing of a Wikipedia Dump on Hadoop.
DenisM 13y ago
So, does it parse the markdown of the text as well? I could not quite get it from the documentation. It would be quite exciting if it could - the markdown is notoriously complex, and creates a lot of problem if all you want is te body of text (e.g. for training a machine learning algorithm).
- matthewarkin 13y ago
  I doubt it (though I haven't tested it). Wikipedia's API though lets you get an HTML version of a page.
- TorKlingberg 13y ago
  Wikipedia articles are not written in markdown but in wikicode. Wikicode is much more complex to parse, but allows more complex formatting. You may be looking for the word 'markup' which is a general term that included markdown, HTML, wikicode, bbcode etc. I doubt this library parses the wikicode. What format would it parse it into? As matthewarkin said, there is a Wikipedia API to convert wikicode into HTML.
  - diegolo 13y ago
    It does that indeed! It will process a field paragraph, containing a list of parsed paragraphs in the article, as simple UTF8 text. Lists and Tables are collected in separated fields, have a look to the Article object. For the wikicode parsing I relied on the JWPL Library http://www.ukp.tu-darmstadt.de/software/jwpl/ http://www.ukp.tu-darmstadt.de/software/jwpl/
fox91 13y ago
Why would I need the dump in JSON? XML is for sure uglier but at least there are efficient SAX parsers.
- dk8996 13y ago
  I would like to know this as well. What is gained by using this tool... converting to JSON?
  - tcdent 13y ago
    JSON is cool and hip. More seriously, it has been gaining popularity because of it's direct relationship to built-in structures in most programming languages. For example: XML doesn't have a concept of arrays, but accomplishes storing collections in it's own special way. It's up to the parser to determine how this should be referenced and interacted-with, while with JSON there's no question: it's an array.
    - sillysaurus2 13y ago
      For example, now I know for a fact I can easily manipulate Wikipedia's data dump using Python thanks to this JSON change. import json js = json.loads(wikidump) That's it. The only reason it's so simple is thanks to this JSON conversion. XML? No clue how to manipulate it properly. I could figure it out of course, but now I don't have to. JSON + Python just works™.
      - diegolo 13y ago
        yes, and also it makes easy to manipulate the dump using Hadoop, and retrieving some values for all the article using jq http://stedolan.github.io/jq/ http://stedolan.github.io/jq/, could you do the same with XML? Btw You can find a little sample of the dump here: https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikipedia-sample.json https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikiped...
diegolo 13y ago
A little sample of the dump: https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikipedia-sample.json https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikiped...

40 points by ikuyamada 13y ago

Some might be interested in https://github.com/andrewvc/wikiparse https://github.com/andrewvc/wikiparse , code I wrote to dump wikipedia into elasticsearch. Here's a chapter in my book on searching it: http://exploringelasticsearch.com/book/searching-natural-language/searching-the-wikipedia-dataset.html http://exploringelasticsearch.com/book/searching-natural-lan...

sktrdie 13y ago

What about DBPedia http://dbpedia.org/page/The_Shining_(film) http://dbpedia.org/page/The_Shining_(film) ? That's available as JSON, and much larger community/effort behind it.

brianshaler 13y ago
There's a much larger community/effort behind DBPedia than Wikipedia? Or are you comparing DBPedia to this utility that converts Wikipedia data dumps? Edit: looks like I need to do some more research into DBPedia. It sounds like that community has created a dataset that categorizes Wikipedia article (e.g. "things" and "facts")
exg 13y ago
Correct me if I am wrong, but I do not think that DBPedia includes the full Wikipedia articles (only extended abstracts). So I believe that json dumps of DBPedia and Wikipedia dumps produced using this tool are two different things.
ikuyamada 13y ago
In my understanding, DBpedia is a project for extracting data mainly from Wikipedia infoboxes (not whole data dumps) by collaboratively creating rules for converting data into cleaner schema that enables to perform a query such as SPARQL. I think this is a project that directly converts wikipedia dump xml to JSON for easier manipulation, which differs from DBpedia.
diegolo 13y ago
Hi, I'm the author of Json-Wikipedia. It's different from DB-Pedia. DBPedia converts the semistructured data containined in the infoboxes in rdf triples, while Json-Wikipedia converts ALL the content of the page (text, infoboxes, links) into a JSON object. I did this because I need some features that were not available on dbpedia (e.g., the whole text of an article, all the anchors referring to an article etc etc). Also Json-Wikipedia simplifies the processing of a Wikipedia Dump on Hadoop.

DenisM 13y ago

So, does it parse the markdown of the text as well? I could not quite get it from the documentation. It would be quite exciting if it could - the markdown is notoriously complex, and creates a lot of problem if all you want is te body of text (e.g. for training a machine learning algorithm).

matthewarkin 13y ago
I doubt it (though I haven't tested it). Wikipedia's API though lets you get an HTML version of a page.
TorKlingberg 13y ago
Wikipedia articles are not written in markdown but in wikicode. Wikicode is much more complex to parse, but allows more complex formatting. You may be looking for the word 'markup' which is a general term that included markdown, HTML, wikicode, bbcode etc. I doubt this library parses the wikicode. What format would it parse it into? As matthewarkin said, there is a Wikipedia API to convert wikicode into HTML.
- diegolo 13y ago
  It does that indeed! It will process a field paragraph, containing a list of parsed paragraphs in the article, as simple UTF8 text. Lists and Tables are collected in separated fields, have a look to the Article object. For the wikicode parsing I relied on the JWPL Library http://www.ukp.tu-darmstadt.de/software/jwpl/ http://www.ukp.tu-darmstadt.de/software/jwpl/

fox91 13y ago

Why would I need the dump in JSON? XML is for sure uglier but at least there are efficient SAX parsers.

dk8996 13y ago
I would like to know this as well. What is gained by using this tool... converting to JSON?
- tcdent 13y ago
  JSON is cool and hip. More seriously, it has been gaining popularity because of it's direct relationship to built-in structures in most programming languages. For example: XML doesn't have a concept of arrays, but accomplishes storing collections in it's own special way. It's up to the parser to determine how this should be referenced and interacted-with, while with JSON there's no question: it's an array.
  - sillysaurus2 13y ago
    For example, now I know for a fact I can easily manipulate Wikipedia's data dump using Python thanks to this JSON change. import json js = json.loads(wikidump) That's it. The only reason it's so simple is thanks to this JSON conversion. XML? No clue how to manipulate it properly. I could figure it out of course, but now I don't have to. JSON + Python just works™.
    - diegolo 13y ago
      yes, and also it makes easy to manipulate the dump using Hadoop, and retrieving some values for all the article using jq http://stedolan.github.io/jq/ http://stedolan.github.io/jq/, could you do the same with XML? Btw You can find a little sample of the dump here: https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikipedia-sample.json https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikiped...

diegolo 13y ago

A little sample of the dump: https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikipedia-sample.json https://dl.dropboxusercontent.com/u/4663256/tmp/json-wikiped...