3 ms·
I had a lightly similar idea that I hope to implement this summer. I mention my idea here in case someone else wants to do it first (and save me the work!). I
by mark_l_watson 11y ago
I had a lightly similar idea that I hope to implement this summer. I mention my idea here in case someone else wants to do it first (and save me the work!).
I keep just the text for the English language articles on my laptop in a text file, one line per article. It is a little more than two gigs. My idea is to compress the text using a ZIP entry for each article, build a Lucene index for local search, add a small Java web app to search and peruse a local copy of Wikipedia (or host it for family and friends) and package it all in one Java JAR file. This could provide a useful local copy of the text in Wikipedia in one handy 4 or 5 gigabyte file.
The downsides to this idea: no media files and it would not be up to date.
Edit: I use the Wikipedia text for NLP experiments, including finding entities (my own code) and processing with both Open Calais and Standford NLP libraries.
- sjs382 11y agoThere are lots of Offline versions of Wikipedia. Some have search, some have media, etc. Here's one: http://xowa.sourceforge.net/ http://xowa.sourceforge.net/
- mark_l_watson 11y agoThanks, that looks good. I might still try my idea for a compressed.text only version but I will try xowa first. Thanks again.
- toomuchtodo 11y agoWith the recently released 512GB microSD cards, you could carry all of wikipedia around (not text only). EDIT: It appears that since 2013, Wikipedia can fit on a 128GB SD card, including images. http://arstechnica.com/information-technology/2013/11/all-of-wikipedia-can-be-installed-to-your-desktop-in-just-30-hours/ http://arstechnica.com/information-technology/2013/11/all-of...
- vmorgulis 11y agoThere is also Kiwix (https://en.wikipedia.org/wiki/Kiwix https://en.wikipedia.org/wiki/Kiwix). It can download recent wikipedia dumps as torrents.
- sp332 11y agoHow do you get the text without all the wiki markup?
- agumonkey 11y agohttp://stackoverflow.com/questions/8555320/is-there-a-clean-wikipedia-api-just-for-retrieve-content-summary http://stackoverflow.com/questions/8555320/is-there-a-clean-... https://www.mediawiki.org/wiki/API:Data_formats https://www.mediawiki.org/wiki/API:Data_formats https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=Stack%20Overflow https://en.wikipedia.org/w/api.php?action=query&prop=extract...
- sp332 11y agoWell most of those answers just replace the MediaWiki markup with HTML markup. At least one of the answers strips the HTML tags out. I already have the MediaWiki version (taken from an XML dump), I was just wondering if there's an easy way to make that useful for NLP projects.