6 ms·

Yet this also exists: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download

dangoodmanUT 2y ago
see you in a year when that finishes
StableAlkyne 2y ago
It does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup. Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.
riku_iki 2y ago
for now
itishappy 2y ago
Wikimedia is Wikipedia's parent entity, but it's also the parent of: MediaWiki Wikibooks Wikidata Wikifunctions Wikimedia Commons Wikinews Wikiquote Wikisource Wikispecies Wikiversity Wikivoyage Wiktionary https://en.wikipedia.org/wiki/Wikimedia_Foundation https://en.wikipedia.org/wiki/Wikimedia_Foundation
- violet13 2y ago
  It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.
  - emw 2y ago
    Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.
    - resolutebat 2y ago
      Wikivoyage is underrated and that was not helped by the acrimonious split with Wikitravel (which was acquired a predatory marketing company), but it finally seems to be pulling ahead.
    - hehdhdjehehegwv 2y ago
      One of my favorite LLM applications is getting them to write wikidata queries. The data is amazing, but the query language is nothing but pure hell.
  - grape_surgeon 2y ago
    Disagree, this is too dismissive. Commons, Wikidata, and Wiktionary are all useful. Especially Wiktionary; probably one of the best online dictionaries imo. Often has a lot of unique info that's hard to find even in dictionaries, very good etymologies. All useful in ML.
paxys 2y ago
Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.
- ks2048 2y ago
  I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.
  - codetrotter 2y ago
    There’s also Wikidata which has machine readable data for everyone https://m.wikidata.org/wiki/Wikidata:Main_Page https://m.wikidata.org/wiki/Wikidata:Main_Page > Wikidata is a free and open knowledge base that can be read and edited by both humans and machines. > Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.
    - riku_iki 2y ago
      Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata
  - bawolff 2y ago
    Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint. Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/ https://query.wikidata.org/
  - 2y ago
mtmail 2y ago
I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).
- chuckreynolds 2y ago
  Wondering then if you've tried the Snapshot API in wikimedia enterprise?
- w10-1 2y ago
  Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service. But quality does take money.
  - madeofpalk 2y ago
    What are the incentives for more paying customers? Wikimedia Foundation, being a non-profit, has a lot less incentives to chase money at the expense of product quality.
    - graemep 2y ago
      The for profit parts are meant to chase money, and higher revenues justify higher management salaries ("I am running a bigger organisation so I deserve to be paid more"). This is widely accepted as being a motive for misaligned incentives in for profit businesses (with regard to takeovers) too.
- boramalper 2y ago
  > I'm running a mirror for OpenStreetMap Curious why you're running a mirror for OSM, is it public or for your private use? If you don't mind, I'd love to learn more about it; sounds super interesting to me.
ImaCake 2y ago
Having the data available for download and a paywalled API is pretty standard practice for a lot of this kind of data. Copernicus weather/satellite data is often setup like this. There's a free download option, a free (limited) API, and then if you wanna get fancy you can use a more feature rich paid API. Many other weather services do similar. It seems to work pretty well with the usual caveats.

dangoodmanUT 2y ago

see you in a year when that finishes

StableAlkyne 2y ago

It does exist, it is just incredibly painful to use -- it's a raw SQL dump of each individual page, which is missing some of the dynamically generated parts of the page you might care about (like categories, unless you separately download that table too). You also have to then run it through wiki software locally if you want the rendered page instead of markup. Don't get me wrong, it is wonderful that the Wikipedia team offers this, and I am grateful they give anything at all for offline usage. It just feels like it's intended more as a side product of their backup process, rather than something you're really supposed to use.

riku_iki 2y ago

for now

itishappy 2y ago

Wikimedia is Wikipedia's parent entity, but it's also the parent of: MediaWiki Wikibooks Wikidata Wikifunctions Wikimedia Commons Wikinews Wikiquote Wikisource Wikispecies Wikiversity Wikivoyage Wiktionary https://en.wikipedia.org/wiki/Wikimedia_Foundation https://en.wikipedia.org/wiki/Wikimedia_Foundation

violet13 2y ago
It sounds like an impressive list, but many of these are ghost towns and are of little value to machine learning. Wikiversity is a mess... Wikipedia is the crown jewel and probably the only thing of unique commercial value for ML.
- emw 2y ago
  Wikipedia indeed seems the most valuable for ML, by far. Wikidata, Wikimedia Commons, and Wiktionary also seem useful there.
  - resolutebat 2y ago
    Wikivoyage is underrated and that was not helped by the acrimonious split with Wikitravel (which was acquired a predatory marketing company), but it finally seems to be pulling ahead.
  - hehdhdjehehegwv 2y ago
    One of my favorite LLM applications is getting them to write wikidata queries. The data is amazing, but the query language is nothing but pure hell.
- grape_surgeon 2y ago
  Disagree, this is too dismissive. Commons, Wikidata, and Wiktionary are all useful. Especially Wiktionary; probably one of the best online dictionaries imo. Often has a lot of unique info that's hard to find even in dictionaries, very good etymologies. All useful in ML.

paxys 2y ago

Which is a good thing. The entire corpus is CC-licensed and anyone can download it for free. If you want a real-time API, performance SLAs, machine parsable formats, support etc. then pay for it.

ks2048 2y ago
I don't have a problem with having paid special services, but the "machine parsable formats" is a bit troubling since I think that should be a core part of the open wikipedia project. I submit this link after coming across this site while Googling for info on parsing wikipedia "infoboxes". I plan to check out their "Article Structured Contents (BETA)" API. Improving infoboxes to be machine-readable seems important. And it would be bad if didn't do this because it's a revenue stream for them.
- codetrotter 2y ago
  There’s also Wikidata which has machine readable data for everyone https://m.wikidata.org/wiki/Wikidata:Main_Page https://m.wikidata.org/wiki/Wikidata:Main_Page > Wikidata is a free and open knowledge base that can be read and edited by both humans and machines. > Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wiktionary, Wikisource, and others.
  - riku_iki 2y ago
    Yes, but it is different dataset, many(majority?) infoboxes are not in wikidata
- bawolff 2y ago
  Just fyi, i think the plan for machine readable infoboxes is stick the data in wikidata, which has its own dumps in machine readable format as well as a publicly accessible SPARQL query endpoint. Its actually quite cool. If you have never played with it i encourage checking out some of the example queries (there is a button labelled examples) on https://query.wikidata.org/ https://query.wikidata.org/
- 2y ago

mtmail 2y ago

I'm using those dumps for a monthly data process. It breaks regularly, mirrors disappear, there's only very few mirrors with the full data and little history. (I'm running a mirror for OpenStreetMap but don't have the disk capacity or bandwith for wikipedia's mirror requirements).

chuckreynolds 2y ago
Wondering then if you've tried the Snapshot API in wikimedia enterprise?
w10-1 2y ago
Right, the incentives are adverse: if they lower quality of service for free offerings, they'll be rewarded with more paying customers for reliable service. But quality does take money.
- madeofpalk 2y ago
  What are the incentives for more paying customers? Wikimedia Foundation, being a non-profit, has a lot less incentives to chase money at the expense of product quality.
  - graemep 2y ago
    The for profit parts are meant to chase money, and higher revenues justify higher management salaries ("I am running a bigger organisation so I deserve to be paid more"). This is widely accepted as being a motive for misaligned incentives in for profit businesses (with regard to takeovers) too.
boramalper 2y ago
> I'm running a mirror for OpenStreetMap Curious why you're running a mirror for OSM, is it public or for your private use? If you don't mind, I'd love to learn more about it; sounds super interesting to me.

ImaCake 2y ago

Having the data available for download and a paywalled API is pretty standard practice for a lot of this kind of data. Copernicus weather/satellite data is often setup like this. There's a free download option, a free (limited) API, and then if you wanna get fancy you can use a more feature rich paid API. Many other weather services do similar. It seems to work pretty well with the usual caveats.