17 ms·
How to download all of Wikipedia onto a USB flash drive
- gbraad 4y agoStill using a WikiReader?
- blue1 4y agoDoes it include the images or it’s just the text?
- 0x073 4y agoYes, with images but only english All possible dumps: https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/
- deleted 4y ago[deleted]
- barbs 4y agoIs there a portable version of Kiwix? Would be cool if you could plug the USB into any computer and start reading Wikipedia without having to install anything.
- tehnicaorg 4y agoYes. You download a zip archive. Unpack from 121MB to 263MB, and start the exe. (assuming you're using Windows)
- orliesaurus 4y agoOh wow, I thought this was gonna be a REALLY large file, but only 95GB not bad, some worthless videogames are larger haha
- aendruk 4y agoCirca 2003 I carried around a pared down copy on a Pocket PC. Dropping a few chosen categories (who needs Sports?) allowed it to barely fit on a 1-GB SD card.
- FeistySkink 4y agoPeople going back in time need sports. An almonac of some kind.
- II2II 4y agoWhile handy, it would be a bit too conspicuous. At least one could claim that an almanac is a novelty print.
- bscphil 4y agoI was curious how they achieve this. It looks like the underlying file format uses LZMA, or optionally Zstd, compression. Both achieve pretty high compression ratios against plain text and markup. > Its file compression uses LZMA2, as implemented by the xz-utils library, and, more recently, Zstandard. The openZIM project is sponsored by Wikimedia CH, and supported by the Wikimedia Foundation. https://en.wikipedia.org/wiki/ZIM_(file_format) https://en.wikipedia.org/wiki/ZIM_(file_format)
- kragen 4y agoThe more important thing is that they aggressively downsize the images and omit the history and talk pages. Even if they were using LZW it would probably only triple the filesize.
- keepquestioning 4y agoI remember the era of stupidly large games.
- orliesaurus 4y ago
- thakoppno 4y agoSomewhere around the original ipad era, I believe there was a curated subset of wikipedia articles that may have been called something like Educator’s Edition. It worked offline and had images and I traveled to Peru with it and learned so much. Does anyone remember this sort of thing? I’ve tried wix formatted copies and they do work but the experience on an offline ipad was simply better. Thanks in advance.
- Rediscover 4y agoYes, I remember - I had a copy on an SD card on my OLPC. I believed it morphed into "Wikipedia for Schools" ^0 - possibly this ^1 is a comment about it? 0: https://en.m.wikipedia.org/wiki/Wikipedia:Wikipedia_for_Schools https://en.m.wikipedia.org/wiki/Wikipedia:Wikipedia_for_Scho... 1: https://www.speedofcreativity.org/2008/11/11/wikipedia-to-go-an-offline-download-for-schools/ https://www.speedofcreativity.org/2008/11/11/wikipedia-to-go...
- thehours 4y agoTangent - I’ve noticed a lot more comments like this using the “^0” syntax for citations vs the traditional “[0]” one I’ve become accustomed to seeing on HN. Is there a real shift happening here and, if so, why?
- ashraful 4y agomaybe: https://github.blog/changelog/2021-09-30-footnotes-now-supported-in-markdown-fields/ https://github.blog/changelog/2021-09-30-footnotes-now-suppo...
- teh_klev 4y agoChecking to see if supported on HN [^1] Edit: nope :) [^1]: https://github.blog/changelog/2021-09-30-footnotes-now-supported-in-markdown-fields/ https://github.blog/changelog/2021-09-30-footnotes-now-suppo...
- pupppet 4y agoCan anyone recommend a hardy device for viewing the content? As nutty as it sounds, in some post-apocalyptic world it would sure be nice to have. I'd keep it under the bed just in case..
- c7b 4y agoHave you looked at e-Ink readers?
- SahAssar 4y agoIf you follow the logic that anything is at about half its life that would probably be an older thinkpad laptop, like an x61 or x200. If you are willing to spend the money on something newer perhaps a thoughbook. I have a modded kobo ebook reader (I upgraded mine to 256GB storage and have project gutenberg, wikipedia and a few other things on it) with a good solar powerbank.
- bscphil 4y ago> If you follow the logic that anything is at about half its life I don't think that makes any sense. By that logic any currently working device should be assumed to last another $currentlifetime. My 20 year old car is not gonna last another 20 years. My 10 year old laptop won't last another 10. If my car somehow did last another 20 years, it would not then make sense to assume it would still be running in another 40. Makes more sense to look at all objects of the same class. If 75% of laptops are dead in 10 years and 95% are dead in 15, and your laptop is 10 years old, you can infer that 5 out of 25 surviving laptops will make it another 5 years, or 20%. (These numbers completely made up, just an example.)
- ScottEvtuch 4y agoI think the idea of "everything is about half its life" is to account for survivorship bias in longevity. The only units that make it to the 95th percentile lifetimes clearly got luckier with parts and can reasonably be expected to last longer.
- colordrops 4y agoIs there a way to keep a mirror that stays in sync?
- doomrobo 4y agoIt looks like Kiwix uses the ZIM file format, which appears to have diffing support [0] (see zimdiff and zimpatch). That said, it doesn't look like Kiwix actually publishes those diffs. [0] https://github.com/openzim/zim-tools/tree/master/src https://github.com/openzim/zim-tools/tree/master/src
- kloch 4y agoI wonder if there is an offline backup of Wikipedia on ISS? There should be. And on every manned space mission.
- Dig1t 4y agoWhy not just every space mission, period?
- vorpalhex 4y agoWell the robots don't read too well..
- Rebelgecko 4y agoHow much would the science capabilities of a telescope like JWST be reduced if 1/3 of its SSD was repurposed for storing the latest wikipedia dump (that 1/3 number is assuming it's only English, compressed, and without images)? To me that seems like an easy cost/benefit analysis.
- autoexec 4y agoHow much would the science capabilities of a telescope like JWST be reduced if we left its SSD alone and just taped a USB drive to the side of it somewhere that contained wikipieda?
- Rebelgecko 4y agoWould duct tape pass the pre-launch vibe check? You'd have to do some engineering work to make sure it's sturdy, doesn't have any impact wrt oscillations, won't create FOD (debris) etc. Once you've done all that work, I'm not sure what you've actually accomplished. By the time any sentient being gets around to visiting JWST, I wouldn't be surprised if an unshielded commercial drive would be rendered totally unusable by radiation.
- bagels 4y agoWhy should there be?
- bombcar 4y agoKiwix is great - I have a collection of various things from their library https://library.kiwix.org/?lang=eng https://library.kiwix.org/?lang=eng downloaded for when I'm on a plane or the internet is otherwise unavailable. That and the TeXlive PDF manuals can get me through anything.
- daneel_w 4y agoI second Kiwix. I found out about it not too long ago on the topic of portable Wikipedia readers. It really stands out as the best software part of such a solution.
- 23B1 4y agoI third Kiwix. Immensely useful when I was deployed without internet.
- kragen 4y agoThat sounds interesting, what was the context?
- 23B1 4y agoHi sorry for the delayed reply, yes I was military and in the early days of Iraq/Afghanistan we didn't have much access to the internet so I brought it with me.
- danuker 4y agoMy guess is military.
- dtgriscom 4y ago
- ivansavz 4y agoLibrary or school in a remote village. There are computers (usually old computers), there might even be a LAN of some sort, but no internet (or very slow internet). In those cases having local access to Wikipedia (and not necessarily just en; Kiwix has archives for all the languages) can be a great learning resource and reference.
- sprash 4y agoIs there something similar for Stack Overflow?
- Jun8 4y agohttps://library.kiwix.org/?lang=eng&category=stack_exchange https://library.kiwix.org/?lang=eng&category=stack_exchange
- deleted 4y ago[deleted]
- ankaAr 4y agoKiwix can do that also. You needs to specify the ZIM file and it works: https://wiki.kiwix.org/wiki/Content_in_all_languages https://wiki.kiwix.org/wiki/Content_in_all_languages Why I know that? I wanted to travel as system administrator in some antartica base with a whole copy of stackoverflow with me.
- kragen 4y agoYes, there are Kiwix .zim files of Stack Overflow. I think Zeal also may have an SO docset.
- int_19h 4y agoKiwix has Stack Overflow (and various StackExchange subsites), Project Gutenberg, TED talks, and plenty more. You can also request something.
- sqrt_1 4y agoArticle mentions to format to exFat as NTFS has a 4GB limit - I don't think that is true.
- deleted 4y ago[deleted]
- Wingman4l7 4y agoIt's not -- FAT32 is the one with the 4GB limit. NTFS has much less native support on Macs than exFAT, though.
- ComputerGuru 4y agoFAT32 can even be used with larger sizes if you just format with a larger cluster size. Since each bundle/shard is 1MB minimum that is not a problem here.
- dmitrygr 4y agoFile size is still limited to 0xffffffff (the dir entry only has 32 bits to store it). Some broken implementations even treat it as signed, and files over 2GB become problematic
- ComputerGuru 4y agoAh sorry, you're right. That allows FAT32 to be used for larger partition sizes, but the file size limit remains in place.
- aaron695 4y ago
- PaulDavisThe1st 4y agoCan someone explain what the role of kiwix in all this, please?
- kragen 4y agoIt provides access to the content of the zimfile and an interface for downloading zimfiles.
- PaulDavisThe1st 4y agoThanks. I had not understood that the download is the actual "raw" wiki files, not the pages as they would be delivered to a browser.
- kragen 4y agoIt's actually a compressed archive, but I think the contents are in fact HTML and other browser-accessible media types.
- deleted 4y ago[deleted]
- londons_explore 4y agoNote that it's possible to make wikipedia substantially smaller if you're happy to use more aggressive compression algorithms. Kiwix divides the data into chunks and adds various indexes and stuff to allow searching data and fast access, even on slow CPU devices. But if you can live with slow loading, you can probably halve the storage space required, or maybe more.
- kragen 4y agoWhat compression algorithms would help? It's already using lzma for the text (in the form of .xz).
- londons_explore 4y agoThe hutter prize is a competition for compressing Wikipedia: http://prize.hutter1.net/ http://prize.hutter1.net/ So the best algorithm to use from there is starlit, with a compression factor of 8.67, compared to lzma in 2MB chunks which can only achieve about 4:1 compression.
- londons_explore 4y agoOh, and if you are happy to wait days or weeks for your compressed data, Fabrice bellards nncp manages even higher ratios (but isn't eligible for the prize because it's too slow)
- planede 4y agoSubmissions for the Hutter price also include the size of the compressor in the "total size". So I assume that's hard to beat if you use huge neural networks on the compression size, even if decompression is fast enough.
- londons_explore 4y agonncp uses neural networks, but 'trains itself' as it goes, so there is no big binary blob involved in the compressor. The only reason it isn't eligible are compute constraints (and I don't think the hutter prize allows a GPU, which nncp needs for any reasonable performance).
- yieldcrv 4y agoprotip: you need to download wikipedia in other languages as well they are not translations, they are completely different articles under the name brand and platform of Wikipedia an entry that may be just a blurb in English may be one of the most comprehensive and fully fleshed out and researched entries on the site in German, for example
- jabbany 4y agoI recall doing such an offline dump with Wikitaxi (https://www.yunqa.de/delphi/apps/wikitaxi/index https://www.yunqa.de/delphi/apps/wikitaxi/index) back when WP was getting banned in China a decade or so ago. IIRC the articles were rather easy to download and convert even on my early 2000s netbook. The media (pictures, video, audio) though was painful to deal with, and it didn't take long to find out that Wikipedia without diagram s and figures was not a great experience.
- kragen 4y agoKiwix's maxi-all Wikipedia zimfiles have pretty much all the pictures that are used in articles, but not the video and audio. And the pictures are too small; often you can't read the text in them.
- breck 4y agoLove it! Imagine if USB Flash drive manufacturers just loaded up new drives with content like this. I mean, why not right? I think the physics means it would even be lighter ;)
- BizarroLand 4y agoEspecially if you didn't know which one you were going to get. Plug it in for a big surprise! (From a verifiable manufacturer who has their customers' happiness and enjoyment at heart)
- sml156 4y agoI for one would not be happy. When I buy a storage device I usually have an intended purpose for that storage and would not like to have to delete all of the files that some manufacturer thought would be useful information but I would have to delete to make room for what I want.
- breck 4y agoSorry I wasn't clear—I 100% agree with you I definitely would generally want a blank one, but would be fun to have some options.
- nneonneo 4y agoCirca 2009 or so, my absolute favorite app for the iPod Touch was Patrick Collison's Offline Wikipedia (yes, that Patrick Collison: https://web.archive.org/web/20100419194443/http://collison.ie/wikipedia-iphone/ https://web.archive.org/web/20100419194443/http://collison.i...). You could download various wikis that had been pre-processed to fit in a very small space - as I recall, the entire English Wikipedia was a mere 2 GB in size. It was simply magical that I could have access to all of Wikipedia anytime, anywhere offline - especially since the iPod Touch could only connect to the Internet via WiFi. It was particularly useful while travelling, since I could load up articles and just read them on the plane. As I recall, there were several clever things that the app did to reduce the size of the dump; many stub/redirect articles were removed, the formatting was pared down to the bare minimum, and it was all compressed quite efficiently to fit in such a small space. Patrick gives more technical detail on an earlier version of the app's homepage: https://web.archive.org/web/20080523222440/http://collison.ie/wikipedia-iphone/ https://web.archive.org/web/20080523222440/http://collison.i...
- NegativeLatency 4y agoYeah that was awesome, so useful/fun being able to go deep on a wikipedia reading session when there wasn't good mobile coverage and data was expensive. In retrospect I do kinda miss _not_ having cell reception on vacations, as it was easier to disconnect from stuff.
- airstrike 4y ago> In retrospect I do kinda miss _not_ having cell reception on vacations, as it was easier to disconnect from stuff. In retrospect I do miss _not_ having the internet or cell service at all. It's part of why I like to watch shows and movies from the 80s or 90s. It's funny because I love technology as much as anyone else on HN, but at the same time my idea of the perfect retired life is one that is almost entirely offline.
- NegativeLatency 4y agoThe tea ceremony of: sit at computer desk, boot up computer, dial up the internet connection etc was pretty fun. Also the relative novelty. (As I check HN from the bathroom, which would’ve been a childhood dream)
- jscipione 4y agoDo not store 96gb of anything on exfat, use ext4 or APFS or zfs or some journaled file system. Does NTFS really have a 4GB file size limit? Structures should match exfat so that part seems suspect to me.
- DrSiemer 4y agoAfaik NTFS max filesize is 256TB
- BizarroLand 4y ago>> Does NTFS really have a 4GB file size limit? No, but FAT32 does. Exfat, on the other hand has a file size limit of 16 exibibytes. That, combined with exfat's cross-platform mounting (NTFS has a lot of limitations in this regard) makes it a superior formatting system for flash based offline file transfer. On a network? Use zfs+ or something.
- int_19h 4y agoThis is the kind of thing that you download once and then never write anything on that media until you decide to refresh the content. In fact, you might as well mount it read-only. A journaling FS wouldn't do anything useful here.
- jamylak 4y agoI recently discovered https://github.com/tatuylonen/wiktextract https://github.com/tatuylonen/wiktextract wiktextract for wiktionary and https://kaikki.org/ https://kaikki.org/ kaikki has the extracts available as links, but it's only en wiktionary for now.
- r3trohack3r 4y agoKiwix is an amazing project. I used a similar approach for https://wikiscroll.blankenship.io https://wikiscroll.blankenship.io 1. kiwix dump 2. unpack to HTML 3. process with cheerio to create json files 4. Create git repo and push to github pages Works well for infinitely scrolling content, it's just Math.random on top of static files. https://github.com/retrohacker/wikiscroll https://github.com/retrohacker/wikiscroll
- Karrot_Kream 4y agoWhat a cool project, thanks! Sounds like something I'd love to waste time on lol.
- tsol 4y agoThis is really cool. Thanks for posting that
- kibwen 4y agoNow I'm curious: if, hypothetically, wikipedia was just backed by a single git repo and every edit was a commit, how big would it be and how long would it take to clone?
- felipelalli 4y agoWhat about the images?
- e-clinton 4y agoDo you think Apple would approve an app that just offlines Wikipedia?
- mikotodomo 4y agoCool. I don't have a USB Flash Drive though.
- sicp-enjoyer 4y ago$15 can help
- garfield322 4y agoThis would be useful to drop into North Korea?
- bubblethink 4y agoThat's exactly what I was thinking of as well. I remember listening to an episode on Darknet Diaries with this theme where dvds and usb drives are a common way to smuggle things into North Korea.
- haolez 4y agoCould I use something like this to train my own GPT that's obsessed with Wikipedia? :)
- deleted 4y ago[deleted]
- smukherjee19 4y agoWow, this is so cool! 95 GB and I can browse the entire Wikipedia offline!? Thanks so much! https://library.kiwix.org/?lang=eng https://library.kiwix.org/?lang=eng I was looking at what other sites are available, and seems there are quite a few. Are there any specific ones apart from Wikipedia that HN readers would recommend?
- nephanth 4y agoIf you end up programming offline, I remember they had a dump of stackoverflow
- jhatemyjob 4y agoIs there something like this that downloads the full edit history as well?
- joshjoseph97 4y ago
- dangrie158 4y agoIn the old days :tm: I remember doing this as well with a 1GB drive ( and room to spare for some mobile apps). Would be interesting to see a graph of usb size easily available vs. Wikipedia dump size.
- Rediscover 4y agoThat would be very interesting. Thanks, I now have another entry on my To Do list.
- quickthrower2 4y agoI wonder if they snapshot Wikipedia for this, or if they stagger it per article to avoid very recent unreviewed edits getting in to such a download (that would say disappear off the site if those were bad edits or vandalism)
- sanroot99 4y agoThey have snapshots ,there is also official Wikipedia torrent links of dumps
- milkshakes 4y agoI want to like Kiwix -- I downloaded Wikipedia AND StackOverflow -- but it keeps crashing every time I try to search for anything on this M1 macbook.
- rbistolfi 4y agoThere is also CDPedia, a project from Python Argentina originally intended for making Wikipedia available in rural schools without Internet connection. https://github.com/PyAr/CDPedia https://github.com/PyAr/CDPedia http://cdpedia.python.org.ar/index.en.es.html http://cdpedia.python.org.ar/index.en.es.html
- armagon 4y agoFYI, the internet archive hosts a ZIM archive that has dumps of wikipedia and many other works. https://archive.org/details/zimarchive https://archive.org/details/zimarchive I wish it was a little more obvious how to search it, or what all the variations mean, but it looks like a valuable resource. It is worth noting that Kiwix works on multiple OSes and on phones and has a wifi hostspot version (that you might run on an raspberry pi, for example). Internet-in-a-box similarly works as a wifi hostspot for ZOM archives. Lastly, it is worth mentioning that there are tools for creating your own ZIM files; it looks like the most straightforward way is to take a static website and use a utility to convert it into one self-contained file.
- Abishek_Muthian 4y agoThanks for sharing, Can you explain a bit more about creating our own ZIM files (or) for archived websites on Internet Archive? I'm looking for a way to archive all the websites from my browser bookmarks and then download them for offline use.
- _a9 4y agoNot related to the OP topic or zim but I was looking into archiving my bookmarks and other content like documentation sites and wikis. I'll list some of the things I ended up using. ArchiveBox[1]: Pretty much a self-hosted wayback machine. It can save websites as plain html, screenshot, text, and some other formats. I have my bookmarks archived in it and have a bookmarklet to easily add new websites to it. If you use the docker-compose you can enable a full-text search backend for an easy search setup. WebRecorder[2]: A browser extension that creates WACZ archives directly in the browser capturing exactly what content you load. I use it on sites with annoying dynamic content that sites like wayback and ArchiveBox wouldn't be able to copy. ReplayWeb[3]: An interface to browse archive types like WARC, WACZ, and HAR. The interface is just like browsing through your browser. It can be self-hosted as well for the full offline experience. browsertrix-crawler[4]: A CLI tool to scrape websites and output to WACZ. Its super easy to run with Docker and I use it to scrape entire blogs and docs for offline use. It uses Chrome to load webpages and has some extra features like custom browser profiles, interactive login, and autoscroll/autoplay. I use the `--generateWACZ` parameter so I can use ReplayWeb to easily browse through the final output. For bookmark and misc webpage archiving then ArchiveBox should be more than enough. Check out this repo for an amazing list of tools and resources https://github.com/iipc/awesome-web-archiving https://github.com/iipc/awesome-web-archiving [1] https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox [2] https://webrecorder.net https://webrecorder.net [3] https://replayweb.page https://replayweb.page [4] https://github.com/webrecorder/browsertrix-crawler https://github.com/webrecorder/browsertrix-crawler
- stewbrew 4y agoHow can someone use so many words to say "use kiwix".
- bori5 4y agoApropos of nothing stumbled upon encyclopedia Britannica the other day, anyone know what’s up with that and if there are any pros to it vs Wikipedia ?
- davidwritesbugs 4y agoI used Britannica while in prison due to the obvious "No Internet". It works well enough: the articles are OK and from authoritative authors unlike many wikipedia pages but I found them a bit lacking in full detail; the main problem is that the range of topics is much much smaller, to the point where it was far less useful for detailed research. For prison, use as a basic reference, it was probably perfectly OK but for more demanding research it's not adequate.
- peter_d_sherman 4y ago>"After reading this article, you’ll be able to save all ~6 million pages of Wikipedia so you can access the sum of human knowledge regardless of internet connection!" [...] >"The current Wikipedia file dump in English is around 95 GB in size. This means you’ll need something like a 128 GB flash drive to accommodate the large file size." Great article! Also, on a related note, there's an interesting philosophical question related to this: Given the task of preserving the most important human knowledge from the Internet and given a certain limited amount of computer storage -- what specific content (which could include text, pictures, web pages, PDFs, videos, technical drawings, etc.) from what sources do you select, and why? ? So first with 100GB (All of Wikipedia is a great choice, btw!) -- but then with only 10GB, then 1GB, then 100MB, then 10MB, then 1MB, etc. -- all the way down to 64K! (about what an early microcomputer could hold on a floppy disk...) What information do you select for each storage amount, and why? ? (Perhaps I should make this a future interview question at my future company!) Anyway, great article!
- iamwil 4y agoI think I'd rather have stack overflow offline, before I'd want wikipedia offline, though.
- sixhobbits 4y agoThe Library page has three identical looking entries, 100gb, 50gb, and 15gb without any explanation about what is or isn't included in each
- _int3_ 4y agoAnyone doing ZIM of news.ycombinator.com ? Once in a week package would be fine. How to make one?
- yCloser 4y agoand now donate to Wikipedia, because you just caused them to pay for 95Gb of (useless) traffic
- snvzz 4y agoThat really isn't how it works.
- CGamesPlay 4y agoCan anyone explain to me how the kiwix library site works? There’s 3 Wikipedia listings that all have the same name, description, language, and author, but seem to have different content. This pattern repeats for the “Wikipedia 0.8” and “Wikipedia 100” sets. One of the latter says that the top 100 pages on Wikipedia require 889 MB? What’s going on here?
- ernst_mulder 4y agoSome time ago I dreamt that I was in an alien space ship for some reason. Still carrying my phone and laptop bag. They were a friendly lot and asked whether or not I would like to charge my laptop. Do you have 220V sockets I asked. They didn't know what that was. So I needed measurements and definitions. An approximate meter, an approximate second. Coulomb was difficult. I woke up and downloaded Wikipedia the next day. Deleted it again later for lack of harddisk space... But next time this happens I will have an USB stick with all the necessary knowledge. The definitions for voltage, current and frequency should however be printed out in case my laptop battery charge is insufficient for accessing the USB stick.
- zeristor 4y agoUsually alien abductions thoughts revolve around an intrusive test to see if you have a chess cheat device inserted, or should that be upserted?
- mellowhype 4y agowould be cool if kiwix came with an auto-update feature, but given the database size, I believe it's difficult to implement.
- SargeDebian 4y agoThis has to be one of the most poorly structured pieces of writing I've seen in a while. It's way too verbose, and on the one hand there are separate sections like: * Getting a flash drive * Formatting a flash drive (which includes a subsection on not formatting it but buying one that's already formatted instead, while there was a separate section just before this one on buying a drive) * Waiting for a file to download At the same time downloading both Wikipedia and Kiwix are in the same section. Then, installing Kiwix is in a section called "You're done" which isn't next to the section on downloading Kiwix.
- deleted 4y ago[deleted]
- bArray 4y agoI think there are better ways to open ZIM files. I've had massive trouble with Kiwix. The old version seems broke beyond repair and the new version is too heavy. ZIMply on branch `version2` has worked pretty well for me [1]. The search works a lot better and it's really nicely formatted. [1] https://github.com/kimbauters/ZIMply/tree/version2 https://github.com/kimbauters/ZIMply/tree/version2
- porbelm 4y ago95 GB? I remember when it was like 2 GB haha
- icod 4y agoBut but if all of Wikipedia fits on a USB drive, what do they need the millions and millions of Dollars for? /s
- krasi0 4y agoAnd if you're only interested in preserving just some Wiki pages, this browser extension with some automation on top will do the perfect job: https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile No affiliation, just a happy user :)
- jokoon 4y agoSo can it remove things like movies and tv shows and other noise? I remember there was some work done to categorize articles like with the Dewey system, but so far, you can't really reduce the size of those exports. Of course it would require a lot of work. Maybe it's already possible to categorize articles of they belong to a "portal". But yeah, it doesn't seem the Wikipedia foundation really care about those kind of problems. To be fair they lack money.
- a3w 4y agoUno card: Reverse! Is TV Tropes available as a single file ZIM download?
- squarefoot 4y agoHow does this scale with the need to update data with time, corrections etc? Having to download everything again doesn't seem that elegant. I think this wold benefit a lot from some form of incremental backup support, that is, download only what was changed since last time. A possible implementation of that could be a bittorrent distributed git-like mirror so that everyone could maintain their local synced one and be able to create its snapshot on removable media on the fly.
- lxgr 4y agoGiven that the ZIM format is highly compressed, I'd assume that any "diff" approach would be computationally quite intensive [1] – on both sides, unless you require clients to apply all patches, which would allow hosting static patch files on the server side. Bandwidth is getting cheaper and cheaper, and arguably if you can afford to get that initial 100 GB Wikipedia dump, you can afford downloading it more than once (and vice versa, if you can download multi-gigabyte differential updates periodically, you can afford the occasional full re-download). One application where I could see it making sense is a related project [2] which streams the Wikipedia over satellite: Initial downloads at this point probably take several days of uninterrupted reception. [1] Google has once implemented a custom binary diff optimized for Chrome updates, but I'm not sure if it still exists. [2] https://en.wikipedia.org/wiki/Othernet https://en.wikipedia.org/wiki/Othernet
- ryanmercer 4y agoBut why? If civilization collapses I'm not going to think "oh, let me consult Wikipedia" I'm going to think "man, this sucks".
- sjducb 4y agoThere is a ZIM file that contains all of stack overflow. Super useful if you have to program without access to the internet.