5 ms·

Okay, that looked a bit ridiculous in the pre-AI era (who needs to download the whole Wikipedia?), but now I can see the sense in it.

_fat_santa 1y ago
Oh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of laptops and you use one of the offline readers to browse it. This is essentially the modern version of having a library of encyclopedias.
- ozmodiar 1y ago
  I'm thinking less about AI training and more about having a source of (reasonably) reliable information from the net, in case AI generated fake images and generated cross referenced texts start making it too difficult to discern real history from malicious rewrites. It's bad enough now, but can get much worse with the proliferation of AI agents.
  - myself248 1y ago
    Pre-2022 Wikipedia dumps will be analyzed by future historians.
    - normie3000 1y ago
      And banned by future governments.
- ThinkingGuy 1y ago
  There's already a project to serve the use case you're describing (school in a disconnected village): Internet in a Box https://internet-in-a-box.org/ https://internet-in-a-box.org/ They provide offline access to Wikipedia, OpenStreetMap, Project Gutenberg, and many other resources.
jsheard 1y ago
Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bots-strain-wikimedia-as-bandwidth-surges-50/ https://arstechnica.com/information-technology/2025/04/ai-bo...
- sdoering 1y ago
  Tragedy of the commons. And that’s why we can’t have nice things. Because people are people. And will always prioritize egotism over respect for the common good.
  - mistrial9 1y ago
    no - when fragile resources are abused by one endpoint out of one hundred thousand others, and the abuse is one hundred thousand times greater.. how is that a condemnation of the "ways" of "all people" .. what is justice?
  - yreg 1y ago
    But we have nice things. Wikipedia can deal with it just fine.
- petercooper 1y ago
  I bet someone like Cloudflare could pull the dataset each day and serve up a plain text/Markdown version of Wikipedia for rounding error levels of spend. I just loaded a random Wikipedia page and it had a weight of 1.5MB in all for what I worked out would be about 30KB of Markdown (i.e. 50x less bandwidth). Of course, the problem then is getting all these scrapers and bots to actually use the alternative, but Wikimedia could potentially redirect suspected clients in that direction..
  - tough 1y ago
    Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk
BoxFour 1y ago
Also helps save Wikipedia if it gets shut down - which might happen!
- hsuduebc2 1y ago
  True. Musk for example is publicly attacking it for spreading "left-wing lies" because in his wiki page there are statements like "He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and promoting conspiracy theories, and affirming antisemitic, racist, and transphobic comments." which are just pure facts. It would be nice to have something like this more decentralized.
  - AStonesThrow 1y ago
    I was perusing some recent discussions on sources with interest. It seems that Wikipedia's intelligentsia have managed to "blacklist" (deprecate or declare "generally unreliable") practically every prominent source of news in the US that is not centrist or leftist. I kid you not; through a process of attrition they've attacked the very reliability and reputation of every source, including Fox News and the like, and they've told editors sitewide that they simply can't be cited as a "Reliable Secondary Source", like at all. I am not sure if that is an accurate assessment of the situation on the ground for mainstream media, but it certainly exposes some real systemic bias. And this is the highest-order and most enduring method of ingraining systemic bias in the project: by weeding out sources with unfavorable viewpoints and perspectives, saying they publish lies and untruth, and being able to prohibit them globally from any use. And I was pondering this state of affairs and just thinking about Karoline Leavitt's press room, and wondering what will the landscape be, if there is precious little intersection between press outlets who may be favorable or deferent to the present administration, and those which are allowed to be cited on Wikipedia? Ouch!
    - zzzeek 1y ago
      It's not Wikipedia's fault that the vast majority of right wing media consists of pure propaganda, disinformation, and lies.
hagbard_c 1y ago
> who needs to download the whole Wikipedia Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to some prepper scenario where connectivity ends at the cave entry because the 'net has ceased to exist. I would like to have a less politically biased online encyclopedia for the latter scenario, it would be a shame to start a new society based on the same bad ideas which brought down the previous one. If ever a politically neutral LLM becomes available that'd be one of the first tasks I'd put it to: point out bias - any bias - in articles, encyclopedias and other 'sources' (yes, I know, WP is not an original source but for this purpose it is) of knowledge.
- moritzwarhier 1y ago
  Is there a "politically neutral" human? And if there was, what could that person reasonably say about politics?
  - amanaplanacanal 1y ago
    I suspect "politically neutral" is a meaningless phrase. It's just a way for people to tar their political opponents by inference. The problem is: even if you report only facts, there is an editorial function in choosing which facts to report, because it is physically impossible to report all facts. So someone can always point to some sort of bias on choosing which facts to report.
    - tough 1y ago
      And when editors have big ad spenders, you bet they won't criticize the hand that feeds them, most of the time
      - moritzwarhier 1y ago
        But what to do about it, in your opinion? How to prevent people with malicious interests from editing?
verandaguy 1y ago
> who needs to download the whole Wikipedia Anyone archiving the site. Wikipedia is, for its faults, one of the best-curated collections of summarized human knowledge, probably in history. Replicating that knowledge helps build data resilience and protect it against all sorts of disasters. I used to seed their monthly data dump torrent for a while.
krick 1y ago
No idea when your pre-AI era begun, but I was much more excited to host Wikipedia locally 15 years ago than I am now.

_fat_santa 1y ago

Oh no even now there is plenty of use for it outside of AI training. Just think of all the schools in villages all around the world that don't have access to the internet or have a very limited connection. I've worked with folks that would setup local "wikipedia servers" for schools so that kids could access Wikipedia via a local network connection. In other setups they just download all of wikipedia to a set of laptops and you use one of the offline readers to browse it. This is essentially the modern version of having a library of encyclopedias.

ozmodiar 1y ago
I'm thinking less about AI training and more about having a source of (reasonably) reliable information from the net, in case AI generated fake images and generated cross referenced texts start making it too difficult to discern real history from malicious rewrites. It's bad enough now, but can get much worse with the proliferation of AI agents.
- myself248 1y ago
  Pre-2022 Wikipedia dumps will be analyzed by future historians.
  - normie3000 1y ago
    And banned by future governments.
ThinkingGuy 1y ago
There's already a project to serve the use case you're describing (school in a disconnected village): Internet in a Box https://internet-in-a-box.org/ https://internet-in-a-box.org/ They provide offline access to Wikipedia, OpenStreetMap, Project Gutenberg, and many other resources.

jsheard 1y ago

Too bad the AI scrapers don't care, and are melting Wikipedia's production servers anyway. https://arstechnica.com/information-technology/2025/04/ai-bots-strain-wikimedia-as-bandwidth-surges-50/ https://arstechnica.com/information-technology/2025/04/ai-bo...

sdoering 1y ago
Tragedy of the commons. And that’s why we can’t have nice things. Because people are people. And will always prioritize egotism over respect for the common good.
- mistrial9 1y ago
  no - when fragile resources are abused by one endpoint out of one hundred thousand others, and the abuse is one hundred thousand times greater.. how is that a condemnation of the "ways" of "all people" .. what is justice?
- yreg 1y ago
  But we have nice things. Wikipedia can deal with it just fine.
petercooper 1y ago
I bet someone like Cloudflare could pull the dataset each day and serve up a plain text/Markdown version of Wikipedia for rounding error levels of spend. I just loaded a random Wikipedia page and it had a weight of 1.5MB in all for what I worked out would be about 30KB of Markdown (i.e. 50x less bandwidth). Of course, the problem then is getting all these scrapers and bots to actually use the alternative, but Wikimedia could potentially redirect suspected clients in that direction..
- tough 1y ago
  Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk

BoxFour 1y ago

Also helps save Wikipedia if it gets shut down - which might happen!

hsuduebc2 1y ago
True. Musk for example is publicly attacking it for spreading "left-wing lies" because in his wiki page there are statements like "He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and promoting conspiracy theories, and affirming antisemitic, racist, and transphobic comments." which are just pure facts. It would be nice to have something like this more decentralized.
- AStonesThrow 1y ago
  I was perusing some recent discussions on sources with interest. It seems that Wikipedia's intelligentsia have managed to "blacklist" (deprecate or declare "generally unreliable") practically every prominent source of news in the US that is not centrist or leftist. I kid you not; through a process of attrition they've attacked the very reliability and reputation of every source, including Fox News and the like, and they've told editors sitewide that they simply can't be cited as a "Reliable Secondary Source", like at all. I am not sure if that is an accurate assessment of the situation on the ground for mainstream media, but it certainly exposes some real systemic bias. And this is the highest-order and most enduring method of ingraining systemic bias in the project: by weeding out sources with unfavorable viewpoints and perspectives, saying they publish lies and untruth, and being able to prohibit them globally from any use. And I was pondering this state of affairs and just thinking about Karoline Leavitt's press room, and wondering what will the landscape be, if there is precious little intersection between press outlets who may be favorable or deferent to the present administration, and those which are allowed to be cited on Wikipedia? Ouch!
  - zzzeek 1y ago
    It's not Wikipedia's fault that the vast majority of right wing media consists of pure propaganda, disinformation, and lies.

hagbard_c 1y ago

> who needs to download the whole Wikipedia Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to some prepper scenario where connectivity ends at the cave entry because the 'net has ceased to exist. I would like to have a less politically biased online encyclopedia for the latter scenario, it would be a shame to start a new society based on the same bad ideas which brought down the previous one. If ever a politically neutral LLM becomes available that'd be one of the first tasks I'd put it to: point out bias - any bias - in articles, encyclopedias and other 'sources' (yes, I know, WP is not an original source but for this purpose it is) of knowledge.

moritzwarhier 1y ago
Is there a "politically neutral" human? And if there was, what could that person reasonably say about politics?
- amanaplanacanal 1y ago
  I suspect "politically neutral" is a meaningless phrase. It's just a way for people to tar their political opponents by inference. The problem is: even if you report only facts, there is an editorial function in choosing which facts to report, because it is physically impossible to report all facts. So someone can always point to some sort of bias on choosing which facts to report.
  - tough 1y ago
    And when editors have big ad spenders, you bet they won't criticize the hand that feeds them, most of the time
    - moritzwarhier 1y ago
      But what to do about it, in your opinion? How to prevent people with malicious interests from editing?

verandaguy 1y ago

> who needs to download the whole Wikipedia Anyone archiving the site. Wikipedia is, for its faults, one of the best-curated collections of summarized human knowledge, probably in history. Replicating that knowledge helps build data resilience and protect it against all sorts of disasters. I used to seed their monthly data dump torrent for a while.

krick 1y ago

No idea when your pre-AI era begun, but I was much more excited to host Wikipedia locally 15 years ago than I am now.