5 ms·
Wikipedia: Database Download
- btbuildem 2y agoFirst time I visited this page was in January 2025
- cynicalsecurity 2y agoOkay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.
- _fat_santa 2y agoOh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of laptops and you use one of the offline readers to browse it. This is essentially the modern version of having a library of encyclopedias.
- ozmodiar 2y agoI'm thinking less about AI training and more about having a source of (reasonably) reliable information from the net, in case AI generated fake images and generated cross referenced texts start making it too difficult to discern real history from malicious rewrites. It's bad enough now, but can get much worse with the proliferation of AI agents.
- myself248 2y agoPre-2022 Wikipedia dumps will be analyzed by future historians.
- normie3000 2y agoAnd banned by future governments.
- ThinkingGuy 2y agoThere's already a project to serve the use case you're describing (school in a disconnected village): Internet in a Box https://internet-in-a-box.org/ https://internet-in-a-box.org/ They provide offline access to Wikipedia, OpenStreetMap, Project Gutenberg, and many other resources.
- jsheard 2y agoToo bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bots-strain-wikimedia-as-bandwidth-surges-50/ https://arstechnica.com/information-technology/2025/04/ai-bo...
- sdoering 2y agoTragedy of the commons. And that’s why we can’t have nice things. Because people are people. And will always prioritize egotism over respect for the common good.
- mistrial9 2y agono - when fragile resources are abused by one endpoint out of one hundred thousand others, and the abuse is one hundred thousand times greater.. how is that a condemnation of the "ways" of "all people" .. what is justice?
- yreg 2y agoBut we have nice things. Wikipedia can deal with it just fine.
- petercooper 2y agoI bet someone like Cloudflare could pull the dataset each day and serve up a plain text/Markdown version of Wikipedia for rounding error levels of spend. I just loaded a random Wikipedia page and it had a weight of 1.5MB in all for what I worked out would be about 30KB of Markdown (i.e. 50x less bandwidth). Of course, the problem then is getting all these scrapers and bots to actually use the alternative, but Wikimedia could potentially redirect suspected clients in that direction..
- tough 2y agoSomeone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk
- BoxFour 2y agoAlso helps save Wikipedia if it gets shut down - which might happen!
- hsuduebc2 2y agoTrue. Musk for example is publicly attacking it for spreading "left-wing lies" because in his wiki page there are statements like "He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and promoting conspiracy theories, and affirming antisemitic, racist, and transphobic comments." which are just pure facts. It would be nice to have something like this more decentralized.
- AStonesThrow 2y agoI was perusing some recent discussions on sources with interest. It seems that Wikipedia's intelligentsia have managed to "blacklist" (deprecate or declare "generally unreliable") practically every prominent source of news in the US that is not centrist or leftist. I kid you not; through a process of attrition they've attacked the very reliability and reputation of every source, including Fox News and the like, and they've told editors sitewide that they simply can't be cited as a "Reliable Secondary Source", like at all. I am not sure if that is an accurate assessment of the situation on the ground for mainstream media, but it certainly exposes some real systemic bias. And this is the highest-order and most enduring method of ingraining systemic bias in the project: by weeding out sources with unfavorable viewpoints and perspectives, saying they publish lies and untruth, and being able to prohibit them globally from any use. And I was pondering this state of affairs and just thinking about Karoline Leavitt's press room, and wondering what will the landscape be, if there is precious little intersection between press outlets who may be favorable or deferent to the present administration, and those which are allowed to be cited on Wikipedia? Ouch!
- zzzeek 2y agoIt's not Wikipedia's fault that the vast majority of right wing media consists of pure propaganda, disinformation, and lies.
- hagbard_c 2y ago> who needs to download the whole Wikipedia Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to some prepper scenario where connectivity ends at the cave entry because the 'net has ceased to exist. I would like to have a less politically biased online encyclopedia for the latter scenario, it would be a shame to start a new society based on the same bad ideas which brought down the previous one. If ever a politically neutral LLM becomes available that'd be one of the first tasks I'd put it to: point out bias - any bias - in articles, encyclopedias and other 'sources' (yes, I know, WP is not an original source but for this purpose it is) of knowledge.
- moritzwarhier 2y agoIs there a "politically neutral" human? And if there was, what could that person reasonably say about politics?
- amanaplanacanal 2y agoI suspect "politically neutral" is a meaningless phrase. It's just a way for people to tar their political opponents by inference. The problem is: even if you report only facts, there is an editorial function in choosing which facts to report, because it is physically impossible to report all facts. So someone can always point to some sort of bias on choosing which facts to report.
- tough 2y agoAnd when editors have big ad spenders, you bet they won't criticize the hand that feeds them, most of the time
- moritzwarhier 2y agoBut what to do about it, in your opinion? How to prevent people with malicious interests from editing?
- verandaguy 2y ago> who needs to download the whole Wikipedia Anyone archiving the site. Wikipedia is, for its faults, one of the best-curated collections of summarized human knowledge, probably in history. Replicating that knowledge helps build data resilience and protect it against all sorts of disasters. I used to seed their monthly data dump torrent for a while.
- krick 2y agoNo idea when your pre-AI era begun, but I was much more excited to host Wikipedia locally 15 years ago than I am now.
- myself248 2y agoYou can also get it as a .zim file for easy offline browsing with Kiwix. The whole enchilada: https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2024-01.zim.torrent https://download.kiwix.org/zim/wikipedia/wikipedia_en_all_ma... Other versions: https://library.kiwix.org/#lang=eng&category=wikipedia https://library.kiwix.org/#lang=eng&category=wikipedia
- ramses0 2y agoI tried this kiwix the other day, it has like a 300mb "essentials" text version that was interesting. This comment was downvoted and instead, it'd better merit a comment as to "why" it wasn't contributing to the discussion?
- JohnKemeny 2y ago> I tried this kiwix the other day, it has like a 300mb "essentials" text version that was interesting. I didn't downvote the comment, but it's not an incredibly deep contribution, is it? If you really wish to contribute, perhaps you can say what the "'essentials' text version" contained and why you found it interesting?
- we0x 2y agoIt is useful software for offline use and emergencies. For those who may not know, apart from wikis, they also offer offline documentation(Linux distros like ArchWiki, libraries, etc.), medical libraries(Medicine Plus, etc.), and Stack Exchange.
- zaggynl 2y agoWill the 2025 zim be available as well?
- unethical_ban 2y agoI am wondering the same thing. I have Jan 2021, Jan 2024... I want to keep a snapshot each year and I wonder why a new one hasn't been generated. I haven't looked for documentation on creating my own zim file.
- thm 2y agoMight also be worthwhile to download a pre ~2023 dump, because Low-background steel.
- foreigner 2y agoLOL that is an amazing analogy, thank you.
- icepat 2y agoMy go-to as well. Pre-war steel.
- froh 2y agoit's not about pre-war. it's about pre-trinity-nuclear tests. which means uncontaminated by atmospheric radioactive isotopes. it happened at the end of ww-ii but that is not the point.
- deleted 2y ago[deleted]
- sebzim4500 2y agoIt's an important distinction because a lot of ships were sunk during WWII.
- icepat 2y agoYes, however it's also an accepted name for it > Low-background steel, also known as pre-war steel and pre-atomic steel, is any steel produced prior to the detonation of the first nuclear bombs in the 1940s and 1950s. https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
- busyant 2y agojust in case anyone is as obtuse as I am, I believe the joke here is that the contents of Wikipedia might be contaminated with AI generated content starting around 2023. you can probably look up low background steel to complete the analogy.
- jasoncartwright 2y agoIf you've got some spare bandwidth & storage then seeding some of the torrents here is a cheap and fun way of helping Wikipedia out. I've served around 20TB of these dumps in the past year. https://meta.wikimedia.org/wiki/Data_dump_torrents https://meta.wikimedia.org/wiki/Data_dump_torrents
- mhitza 2y agoDo you happen to know why wikipedia didn't embrace torrents as the default download method?
- davidkwast 2y agoI think it is a nice use case for IPFS
- dijit 2y agoSpeculating: because torrents are not especially good at dealing with small modifications? Most people probably won't seed many versions, so it's a losing effort, and you need to allocate a huge chunk of space for each version. Deduplicating filesystems are sadly not in vogue.
- mmooss 2y agoMost people don't use torrents.
- albert_e 2y agoThere are non English versions of Wikipedia also. Can anyone please point to information on how we can download a copy of one specific language version?
- detaro 2y agothe linked page tells you where those are available already
- JKCalhoun 2y agoYou can find plenty of other languages here [1] for example. These are the Kiwix versions. [1] https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/
- haunter 2y agoIt's literally described there on the page if you open it...
- JKCalhoun 2y agoI more or less do this every year — grab the latest Kiwix, English version (about 100 GB or so). I keep the older ones as well.
- JKCalhoun 2y agoIs there a RAG for Wikipedia? I may not be using the term correctly here. In short, I would love a local LLM + Wikipedia snapshot so that I can have an offline, self-hosted ... Hitchhiker's Guide to Earth.
- maronato 2y agoHuggingface has a few datasets of Wikipedia embeddings. Here’s a few results: https://huggingface.co/search/full-text?q=Wikipedia+embeddings&type=dataset https://huggingface.co/search/full-text?q=Wikipedia+embeddin... And the first result, which is probably what you’ll want to use: https://huggingface.co/datasets/Upstash/wikipedia-2024-06-bge-m3/blob/main/README.md https://huggingface.co/datasets/Upstash/wikipedia-2024-06-bg... I recommend you go for pgvector or a similar self hosted solution to calculate the similarities instead of a service like Vector.
- miki123211 2y agoI wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them in hard-to-reach locations around the world, to make sure at least one survives whatever calamity might befall us in the future.
- willis936 2y agoYou could even make it radiation tolerant by printing it.
- nine_k 2y agoBe certain to use acid-free paper [1]. The typical cheap bright-white paper of today will have hard time staying in a good condition in 100-200 years. Ideally go for the ultra-durable cotton-rag paper used e.g. for paper money. [1]: https://en.wikipedia.org/wiki/Acid-free_paper https://en.wikipedia.org/wiki/Acid-free_paper
- lanna 2y agoRelevant XKCD: https://what-if.xkcd.com/59/ https://what-if.xkcd.com/59/
- jjeaff 2y agoThen figure out how and where to store the over one thousand volumes that have 1200 pages each.
- interroboink 2y agoFrom "Plenty of Room at the Bottom"[1]: What would happen if I print all this down at the scale we have been discussing? How much space would it take? It would take, of course, the area of about a million pinheads ... All of the information which all of mankind has every recorded in books can be carried around in a pamphlet in your hand — and not written in code, but a simple reproduction of the original pictures, engravings, and everything else on a small scale without loss of resolution. Need a good magnifying glass, though (: [1] https://web.pa.msu.edu/people/yang/RFeynman_plentySpace.pdf https://web.pa.msu.edu/people/yang/RFeynman_plentySpace.pdf
- ande-mnoc 2y agoA bit of warning to people who use the database download: the dumps don’t reflect a consistent state of the database and contain broken and missing data. https://meta.wikimedia.org/wiki/Data_dumps/What_the_dumps_are_not https://meta.wikimedia.org/wiki/Data_dumps/What_the_dumps_ar...
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]