6 ms·

Offline Wikipedia

27 points by Gibbon 17y ago

electromagnetic 17y ago
Wikipedia should really release a mini-portable hard drive or USB stick that contains the majority of Wikipedia and is capable of auto-update. I believe it would be a great source of revenue for them and enable them to run a better encyclopaedia with less hits on their website, especially for frequently accessed sites. I believe full integration of the offline and online would be required. Example: Open the Dilbert wikipedia page, you can browse through to what you want to see. If you have an internet connection you can hit the refresh button (or an update button) and it'll quickly check and load an updated page and update the offline.
- deleted 17y ago
  [deleted]
- pyre 17y ago
  The real problem is that Wikipedia is huge. Just the text+history of the English version is like 16GB of XML to download. And that's not counting the media (images,etc) which is somewheres around 200GB+. I've often thought of creating some sort of offline Wikipedia, but it seems like too complex of a task to undertake in limited free time. To mitigate the huge-ness that is the full Wikipedia, an app would have to take a somewhat more limited approach. It would have to start off with a Wikipedia snapshot (i.e. what the current page of each article looks like right now) sans media. When a user visited a page, the user would have several options (other than reading the page): [1] Download all media for the page (and store for future offline usage) [2] Update the page to the current version (with a sub-option to do media as well) [3] Download the full history of the page (with sub-option for with/without media). You could obviously get a lot more complicated (i.e. 'set this page for automatic updates at a set interval', 'download the full history, but only download the media for the current revision', 'automatically check for internet connection/update page to current revision upon accessing the article',etc) I'm surprised that there hasn't been any similar project. Probably due to some of the complexity vs just having a 'latest snapshot of the article text' and leaving it at that. Due to the size of Wikipedia, updating particular articles on a demand basis is probably the way to go. (Keeping past downloaded revisions of the page would be a design decision due to possibly conflicts with a 'download entire history' option)
  - electromagnetic 17y ago
    I agree it is a complex operation sheerly due to size. I'd say a condensed version (contained on say a 16/32GB USB drive) should contain at least the top 1,000 articles in their entirety for their 12 main categories: Astronomy, Biology, Chemistry, Film and television, Genetics, Geography, Internet, Language, Law, Libraries, Music and People. A fair percentage of the drive is filled with text-only versions of the next 9,000 top articles in the 12 main categories. I'd say 30%, if possible, should be left for user added articles (using a history ranking service for registered users). I'd say a standard version (contained on say a 120GB mini-portable HDD, say iPhone sized, or an equivalently affordable sized drive) should contain the top 5,000 of each category in their entirety and it should be possible to store the rest of the database text-only. A deluxe version (contained on say a 500GB portable HDD) should contain the entirety of Wikipedia, if capable. I think a large part of an offline Wikipedia would require some effort by wikipedia itself to aggregate page visits for thousands of pages. The core pages should be auto-updated whenever an internet connection is available (with user definable choices if a user doesn't want an auto-update, say when they're using a dial-up or a 3G stick). Updates could easily be zipped to compact text updates for the main pages; picture updates should be rare. Wikipedia would also need authorized versions of pages, so a user doesn't need the histories clogging up their drive, which would mean a requirement on a lot of trusted editors and an editorial process. Someone should never be in the middle of nowhere and check their offline wikipedia to find LOLZ as the only thing in an important article. I think the hardware and software can likely be done, I believe it would just require a lot of effort on part of Wikipedia itself. Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access.
    - pyre 17y ago
      > Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access. Not necessarily. The problem is that if I wanted to do a full offline version of Wikipedia (sans media) there are no incremental updates. IIRC, you would have to re-download the whole thing, even for articles with no changes (or minor/insignificant changes). That's what makes the 'on demand' aspect of a 3rd-party offline Wikipedia app the best model from a bandwidth conservation point of view. I agree that Wikipedia could do something like this with at least the text-only portions of the site. I feel that they would run into a lot of legal troubles with the media though. Lots of the media is user-submitted, but things like logos, trademarks, 'fair-use' of copyrighted material could get them into a lot of legal battles if they tried to sell/distribute an offline version. Even with the text portions of the site, I'm sure they would end up with a few lawsuits from large contributors here and there. Even if the lawsuits were frivolous. Even on an official Wikipedia product, you'll have to think about the bandwidth consumption that would happen from these devices phoning home for updates. While it might help them cut down on actual page hits to the site, it might not as people will probably have to access this from a laptop, and would still use the web browser on their cell/smart phone to do a quick lookup on a spur of the moment topic.
- hughprime 17y ago
  I believe it would be a great source of revenue for them and enable them to run a better encyclopaedia with less hits on their website, especially for frequently accessed sites How many people are really likely to buy this, though? With wireless internet from your phone from damn near any populated point on the planet becoming more and more widespread, I can't imagine too many circumstances where this would be useful enough to pay money for.
  - electromagnetic 17y ago
    3G phones only have ~14% market integration, cell phones aren't universally owned, actually some countries have ridiculously low adoption rates for their internet infrastructure. 3G infrastructure is severely lacking compared to 2G infrastructure. There's plenty of cities here in Canada without any 3G access and have over 100,000 people and no current plans for 3G. Even areas with 3G, rarely have usable 3G everywhere, especially around cottage country here in Ontario. The valleys seriously hamper even 2G access, especially anywhere near the water. The majority of people don't live in metropolitan areas, so I believe this would sell hugely. I also believe most people don't want to pay for internet twice. Edit: http://www.coveragemaps.com/gsmposter_world.htm http://www.coveragemaps.com/gsmposter_world.htm Yellow is 3G. It certainly isn't expansive access in NA and considering offline access would likely only be required outside of major populated areas, this generally means outside of 3G areas. Offline is kind of the point of an offline wikipedia.
  - dkasper 17y ago
    I think the Kindle is a good counterexample to your argument. In fact now that I think about it, if you could get Wikipedia on the Kindle that might be a huge selling point.
- deleted 17y ago
  [deleted]
naz 17y ago
The iPhone app Encyclopedia downloads a Wikipedia dump to the phone and works offline. It manages to strip it all down to about 3GB http://www.steamheavyindustries.com http://www.steamheavyindustries.com
- fizx 17y ago
  I wonder if there's a market for a usb drive that contains webkit, wikipedia, and the ability to transparently download deltas when connected. 4GB USB is <$10, sell for $20, viola!
rico37 17y ago
Here's another offline Wikipedia solution: http://aarddict.org http://aarddict.org
- deleted 17y ago
  [deleted]
pyre 17y ago
> Like every other nixy package, type sudo ./configure, sudo make, and sudo make install to get it cooking. Arg. You don't need root/sudo to run './configure' or 'make.' You just need it for 'make install' so that you can install it to /usr/ or /usr/local. And you don't even need it for 'make install' if you are installing to a directory that you have rights to (e.g. PREFIX=~ or PREFIX=~/local). Copious usage of sudo seems to abound nowadays. [Note: disregard this if you do something like wget/unpack all your sources files into someplace like /usr/src]
paraschopra 17y ago
This article might just save me. I agree decompressing Wikipedia takes a long time. On my machine, it took almost 24 hours. More time to decompress than actually downloading it!
webignition 17y ago
Would a site specific browser with some clever caching not be a good compromise? A SSB could start out as an offline Wikipedia reader with no offline content. As and when online content is retrieved it is cached locally forever. When an article is to be viewed the local copy is displayed. If there is an Internet connection the Wikipedia reader can utilise various HTTP mechanisms to determine if the local copy is stale, updating the local copy as required. As a basic MVP that could work, albeit with an initial heavy reliance on an Internet connection. As and when problems with this model are identified they can be addressed and solved. For example, it may be beneficial for the reader, when online, to retrieve not only the currently-requested article but also linked articles up to X links deep. Furthermore, it may be feasible to intelligently analyse a user's usage history and retrieve further articles, whilst online, such that relevant content is more likely to be available to the user when next offline. To begin with, using such an application would be little different from using the online Wikipedia. Over time the local cache would grow and the application's reliance on an Internet connection would decrease. Could this be a viable opensource project?
- eru 17y ago
  You can start with a seed of popular pages instead of completely blank.

27 points by Gibbon 17y ago

Wikipedia should really release a mini-portable hard drive or USB stick that contains the majority of Wikipedia and is capable of auto-update. I believe it would be a great source of revenue for them and enable them to run a better encyclopaedia with less hits on their website, especially for frequently accessed sites. I believe full integration of the offline and online would be required. Example: Open the Dilbert wikipedia page, you can browse through to what you want to see. If you have an internet connection you can hit the refresh button (or an update button) and it'll quickly check and load an updated page and update the offline.

deleted 17y ago
[deleted]
pyre 17y ago
The real problem is that Wikipedia is huge. Just the text+history of the English version is like 16GB of XML to download. And that's not counting the media (images,etc) which is somewheres around 200GB+. I've often thought of creating some sort of offline Wikipedia, but it seems like too complex of a task to undertake in limited free time. To mitigate the huge-ness that is the full Wikipedia, an app would have to take a somewhat more limited approach. It would have to start off with a Wikipedia snapshot (i.e. what the current page of each article looks like right now) sans media. When a user visited a page, the user would have several options (other than reading the page): [1] Download all media for the page (and store for future offline usage) [2] Update the page to the current version (with a sub-option to do media as well) [3] Download the full history of the page (with sub-option for with/without media). You could obviously get a lot more complicated (i.e. 'set this page for automatic updates at a set interval', 'download the full history, but only download the media for the current revision', 'automatically check for internet connection/update page to current revision upon accessing the article',etc) I'm surprised that there hasn't been any similar project. Probably due to some of the complexity vs just having a 'latest snapshot of the article text' and leaving it at that. Due to the size of Wikipedia, updating particular articles on a demand basis is probably the way to go. (Keeping past downloaded revisions of the page would be a design decision due to possibly conflicts with a 'download entire history' option)
- electromagnetic 17y ago
  I agree it is a complex operation sheerly due to size. I'd say a condensed version (contained on say a 16/32GB USB drive) should contain at least the top 1,000 articles in their entirety for their 12 main categories: Astronomy, Biology, Chemistry, Film and television, Genetics, Geography, Internet, Language, Law, Libraries, Music and People. A fair percentage of the drive is filled with text-only versions of the next 9,000 top articles in the 12 main categories. I'd say 30%, if possible, should be left for user added articles (using a history ranking service for registered users). I'd say a standard version (contained on say a 120GB mini-portable HDD, say iPhone sized, or an equivalently affordable sized drive) should contain the top 5,000 of each category in their entirety and it should be possible to store the rest of the database text-only. A deluxe version (contained on say a 500GB portable HDD) should contain the entirety of Wikipedia, if capable. I think a large part of an offline Wikipedia would require some effort by wikipedia itself to aggregate page visits for thousands of pages. The core pages should be auto-updated whenever an internet connection is available (with user definable choices if a user doesn't want an auto-update, say when they're using a dial-up or a 3G stick). Updates could easily be zipped to compact text updates for the main pages; picture updates should be rare. Wikipedia would also need authorized versions of pages, so a user doesn't need the histories clogging up their drive, which would mean a requirement on a lot of trusted editors and an editorial process. Someone should never be in the middle of nowhere and check their offline wikipedia to find LOLZ as the only thing in an important article. I think the hardware and software can likely be done, I believe it would just require a lot of effort on part of Wikipedia itself. Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access.
  - pyre 17y ago
    > Someone just attempting an offline wikipedia on their own HDD is always going to have to go to serious complexities unless there is some form of back system in place to access. Not necessarily. The problem is that if I wanted to do a full offline version of Wikipedia (sans media) there are no incremental updates. IIRC, you would have to re-download the whole thing, even for articles with no changes (or minor/insignificant changes). That's what makes the 'on demand' aspect of a 3rd-party offline Wikipedia app the best model from a bandwidth conservation point of view. I agree that Wikipedia could do something like this with at least the text-only portions of the site. I feel that they would run into a lot of legal troubles with the media though. Lots of the media is user-submitted, but things like logos, trademarks, 'fair-use' of copyrighted material could get them into a lot of legal battles if they tried to sell/distribute an offline version. Even with the text portions of the site, I'm sure they would end up with a few lawsuits from large contributors here and there. Even if the lawsuits were frivolous. Even on an official Wikipedia product, you'll have to think about the bandwidth consumption that would happen from these devices phoning home for updates. While it might help them cut down on actual page hits to the site, it might not as people will probably have to access this from a laptop, and would still use the web browser on their cell/smart phone to do a quick lookup on a spur of the moment topic.
hughprime 17y ago
I believe it would be a great source of revenue for them and enable them to run a better encyclopaedia with less hits on their website, especially for frequently accessed sites How many people are really likely to buy this, though? With wireless internet from your phone from damn near any populated point on the planet becoming more and more widespread, I can't imagine too many circumstances where this would be useful enough to pay money for.
- electromagnetic 17y ago
  3G phones only have ~14% market integration, cell phones aren't universally owned, actually some countries have ridiculously low adoption rates for their internet infrastructure. 3G infrastructure is severely lacking compared to 2G infrastructure. There's plenty of cities here in Canada without any 3G access and have over 100,000 people and no current plans for 3G. Even areas with 3G, rarely have usable 3G everywhere, especially around cottage country here in Ontario. The valleys seriously hamper even 2G access, especially anywhere near the water. The majority of people don't live in metropolitan areas, so I believe this would sell hugely. I also believe most people don't want to pay for internet twice. Edit: http://www.coveragemaps.com/gsmposter_world.htm http://www.coveragemaps.com/gsmposter_world.htm Yellow is 3G. It certainly isn't expansive access in NA and considering offline access would likely only be required outside of major populated areas, this generally means outside of 3G areas. Offline is kind of the point of an offline wikipedia.
- dkasper 17y ago
  I think the Kindle is a good counterexample to your argument. In fact now that I think about it, if you could get Wikipedia on the Kindle that might be a huge selling point.
deleted 17y ago
[deleted]

naz 17y ago

The iPhone app Encyclopedia downloads a Wikipedia dump to the phone and works offline. It manages to strip it all down to about 3GB http://www.steamheavyindustries.com http://www.steamheavyindustries.com

fizx 17y ago
I wonder if there's a market for a usb drive that contains webkit, wikipedia, and the ability to transparently download deltas when connected. 4GB USB is <$10, sell for $20, viola!

rico37 17y ago

Here's another offline Wikipedia solution: http://aarddict.org http://aarddict.org

deleted 17y ago
[deleted]

pyre 17y ago

> Like every other nixy package, type sudo ./configure, sudo make, and sudo make install to get it cooking. Arg. You don't need root/sudo to run './configure' or 'make.' You just need it for 'make install' so that you can install it to /usr/ or /usr/local. And you don't even need it for 'make install' if you are installing to a directory that you have rights to (e.g. PREFIX=~ or PREFIX=~/local). Copious usage of sudo seems to abound nowadays. [Note: disregard this if you do something like wget/unpack all your sources files into someplace like /usr/src]

paraschopra 17y ago

This article might just save me. I agree decompressing Wikipedia takes a long time. On my machine, it took almost 24 hours. More time to decompress than actually downloading it!

webignition 17y ago

Would a site specific browser with some clever caching not be a good compromise? A SSB could start out as an offline Wikipedia reader with no offline content. As and when online content is retrieved it is cached locally forever. When an article is to be viewed the local copy is displayed. If there is an Internet connection the Wikipedia reader can utilise various HTTP mechanisms to determine if the local copy is stale, updating the local copy as required. As a basic MVP that could work, albeit with an initial heavy reliance on an Internet connection. As and when problems with this model are identified they can be addressed and solved. For example, it may be beneficial for the reader, when online, to retrieve not only the currently-requested article but also linked articles up to X links deep. Furthermore, it may be feasible to intelligently analyse a user's usage history and retrieve further articles, whilst online, such that relevant content is more likely to be available to the user when next offline. To begin with, using such an application would be little different from using the online Wikipedia. Over time the local cache would grow and the application's reliance on an Internet connection would decrease. Could this be a viable opensource project?

eru 17y ago
You can start with a seed of popular pages instead of completely blank.