4 ms·

Parsing All of Wikipedia to an Offline Encyclopedia

25 points by kxrm 6y ago

MeinBlutIstBlau 6y ago
This is pretty cool actually. The one thing I always hated about wikipedias .zim files is that they're not really a standard I can use elsewhere. Although I'm having trouble understanding the sorting structure with just one json file I'm looking at. I realize it's sorting by the ID, but is that just simply based off of when the ID in wikipedia was created? However, I think the json file is extraordinarily nice to have because it's all of wikipedia in just a simple text format. If we could get wikipedia as an entire package with all the html, css, javascript but capable of being run locally, I think that would be an amazing game changer using wikipedia offline, especially on a pi! While kiwix "works" in some ways, it also has some major issues that makes using it such a pain. I'd like for wikipedia to be just a completely source of data and information that I could search and pull up just even a basic html file and read that article from looking it up just in my own directory. Overall though this is a super cool project!
- m-p-3 6y ago
  It's not a hugely useful bit of information, but I found out a while ago that most Kobo eReaders are able to handle ZIM files. And some of them can have their "on-board" storage increased, which is actually a normally non-user-accessible SD card. I wouldn't do it on a water-resistant model to avoid breaking the seal.
bicepjai 6y ago
If someone got the entire Wikipedia offline, please do post the size of it.
- tutfbhuf 6y ago
  The author uploaded his plaintext dump here: https://www.kaggle.com/ltcmdrdata/plain-text-wikipedia-202011 https://www.kaggle.com/ltcmdrdata/plain-text-wikipedia-20201... it is 22 GB in 605 .json files.
paulyy_y 6y ago
This seems like a convoluted method to achieve the end goal, straight up regex parsing pages seems like a last ditch effort not the up front start.
- splitbrain 6y ago
  yeah, AFAIK the MediaWiki people spent a lot of time to write a new parser for their visual editor that creates an AST from the syntax. That should be a much better way to create whatever format you want. OTOH this is just hearsay and I never really looked into it. So maybe this was the last ditch and OP just did not mention the failed attempts before?
  - 20after4 6y ago
    tldr; There are 3 parsers for wikitext, 1. the "old" wikitext parser written in php 2. a newer effort written in js and named parsoid 3. parsoid.php which is the one currently in use on (most?) WikiMedia sites. All the important details are on the wiki page: https://www.mediawiki.org/wiki/Parsoid https://www.mediawiki.org/wiki/Parsoid Disclosure: I work for the Wikimedia foundation but not directly on the parsing code. Anything I post on hacker news does not represent the views of my employer.
tutfbhuf 6y ago
Maybe one could create a full text search for Wikipedia out of this data. This would also allow us to easily gather statistical information from Wikipedia, e.g. how often does some word occur in Wikipedia. It would be great to have a Wikipedia to play and query around with.
CTOSian 6y ago
1st thing on my mind is that handheld Openmoko WikiReader
dirtnugget 6y ago
A few years back I remember having had a DVD with all of Wikipedia from a magazine

25 points by kxrm 6y ago

This is pretty cool actually. The one thing I always hated about wikipedias .zim files is that they're not really a standard I can use elsewhere. Although I'm having trouble understanding the sorting structure with just one json file I'm looking at. I realize it's sorting by the ID, but is that just simply based off of when the ID in wikipedia was created? However, I think the json file is extraordinarily nice to have because it's all of wikipedia in just a simple text format. If we could get wikipedia as an entire package with all the html, css, javascript but capable of being run locally, I think that would be an amazing game changer using wikipedia offline, especially on a pi! While kiwix "works" in some ways, it also has some major issues that makes using it such a pain. I'd like for wikipedia to be just a completely source of data and information that I could search and pull up just even a basic html file and read that article from looking it up just in my own directory. Overall though this is a super cool project!

m-p-3 6y ago
It's not a hugely useful bit of information, but I found out a while ago that most Kobo eReaders are able to handle ZIM files. And some of them can have their "on-board" storage increased, which is actually a normally non-user-accessible SD card. I wouldn't do it on a water-resistant model to avoid breaking the seal.

bicepjai 6y ago

If someone got the entire Wikipedia offline, please do post the size of it.

tutfbhuf 6y ago
The author uploaded his plaintext dump here: https://www.kaggle.com/ltcmdrdata/plain-text-wikipedia-202011 https://www.kaggle.com/ltcmdrdata/plain-text-wikipedia-20201... it is 22 GB in 605 .json files.

paulyy_y 6y ago

This seems like a convoluted method to achieve the end goal, straight up regex parsing pages seems like a last ditch effort not the up front start.

splitbrain 6y ago
yeah, AFAIK the MediaWiki people spent a lot of time to write a new parser for their visual editor that creates an AST from the syntax. That should be a much better way to create whatever format you want. OTOH this is just hearsay and I never really looked into it. So maybe this was the last ditch and OP just did not mention the failed attempts before?
- 20after4 6y ago
  tldr; There are 3 parsers for wikitext, 1. the "old" wikitext parser written in php 2. a newer effort written in js and named parsoid 3. parsoid.php which is the one currently in use on (most?) WikiMedia sites. All the important details are on the wiki page: https://www.mediawiki.org/wiki/Parsoid https://www.mediawiki.org/wiki/Parsoid Disclosure: I work for the Wikimedia foundation but not directly on the parsing code. Anything I post on hacker news does not represent the views of my employer.

tutfbhuf 6y ago

Maybe one could create a full text search for Wikipedia out of this data. This would also allow us to easily gather statistical information from Wikipedia, e.g. how often does some word occur in Wikipedia. It would be great to have a Wikipedia to play and query around with.

CTOSian 6y ago

1st thing on my mind is that handheld Openmoko WikiReader

dirtnugget 6y ago

A few years back I remember having had a DVD with all of Wikipedia from a magazine