2 ms·
The real problem is that Wikipedia is huge. Just the text+history of the English version is like 16GB of XML to download. And that's not counting the media (ima
by pyre 17y ago
The real problem is that Wikipedia is huge. Just the text+history of the English version is like 16GB of XML to download. And that's not counting the media (images,etc) which is somewheres around 200GB+.
I've often thought of creating some sort of offline Wikipedia, but it seems like too complex of a task to undertake in limited free time.
To mitigate the huge-ness that is the full Wikipedia, an app would have to take a somewhat more limited approach. It would have to start off with a Wikipedia snapshot (i.e. what the current page of each article looks like right now) sans media. When a user visited a page, the user would have several options (other than reading the page): [1] Download all media for the page (and store for future offline usage) [2] Update the page to the current version (with a sub-option to do media as well) [3] Download the full history of the page (with sub-option for with/without media). You could obviously get a lot more complicated (i.e. 'set this page for automatic updates at a set interval', 'download the full history, but only download the media for the current revision', 'automatically check for internet connection/update page to current revision upon accessing the article',etc)
I'm surprised that there hasn't been any similar project. Probably due to some of the complexity vs just having a 'latest snapshot of the article text' and leaving it at that. Due to the size of Wikipedia, updating particular articles on a demand basis is probably the way to go. (Keeping past downloaded revisions of the page would be a design decision due to possibly conflicts with a 'download entire history' option)
- electromagnetic 17y agoI agree it is a complex operation sheerly due to size. I'd say a condensed version (contained on say a 16/32GB USB drive) should contain at least the top 1,000 articles in their entirety for their 12 main categories: Astronomy, Biology, Chemistry, Film and television, Genetics, Geography, Internet, Language, Law, Libraries, Music and People. A fair percentage of the drive is filled with text-only versions of the next 9,000 top articles in the 12 main categories. I'd say 30%, if possible, should be left for user added articles (using a history ranking service for registered users). I'd say a standard version (contained on say a 120GB mini-portable HDD, say iPhone sized, or an equivalently affordable sized drive) should contain the top 5,000 of each category in their entirety and it should be possible to store the rest of the database text-only. A deluxe version (contained on say a 500GB portable HDD) should contain the entirety of Wikipedia, if capable. I think a large part of an offline Wikipedia would require some effort by wikipedia itself to aggregate page visits for thousands of pages. The core pages should be auto-updated whenever an internet connection is available (with user definable choices if a user doesn't want an auto-update, say when they're using a dial-up or a 3G stick). Updates could easily be zipped to compact text updates for the main pages; picture updates should be rare. Wikipedia would also need authorized versions of pages, so a user doesn't need the histories clogging up their drive, which would mean a requirement on a lot of trusted editors and an editorial process. Someone should never be in the middle of nowhere and check their offline wikipedia to find LOLZ as the only thing in an important article. I think the hardware and software can likely be done, I believe it would just require a lot of effort on part of Wikipedia itself. Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access.
- pyre 17y ago> Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access. Not necessarily. The problem is that if I wanted to do a full offline version of Wikipedia (sans media) there are no incremental updates. IIRC, you would have to re-download the whole thing, even for articles with no changes (or minor/insignificant changes). That's what makes the 'on demand' aspect of a 3rd-party offline Wikipedia app the best model from a bandwidth conservation point of view. I agree that Wikipedia could do something like this with at least the text-only portions of the site. I feel that they would run into a lot of legal troubles with the media though. Lots of the media is user-submitted, but things like logos, trademarks, 'fair-use' of copyrighted material could get them into a lot of legal battles if they tried to sell/distribute an offline version. Even with the text portions of the site, I'm sure they would end up with a few lawsuits from large contributors here and there. Even if the lawsuits were frivolous. Even on an official Wikipedia product, you'll have to think about the bandwidth consumption that would happen from these devices phoning home for updates. While it might help them cut down on actual page hits to the site, it might not as people will probably have to access this from a laptop, and would still use the web browser on their cell/smart phone to do a quick lookup on a spur of the moment topic.