3 ms·
>Extracting just raw text from the Wikipedia dumps is pretty hellish. Did you try just asking them? I'm sure someone there can flick a switch and get it all t
by logicallee 10y ago
>Extracting just raw text from the Wikipedia dumps is pretty hellish.
Did you try just asking them? I'm sure someone there can flick a switch and get it all to you. After all, it's just text, and they must have it all set up to serve however they want.
The total size of the English Wikipedia is under 100 GB as uncompressed xml[1] (bearing in mind that xml is pretty verbose).
This should not be prohibitive for a Wikipedia admin to generate and to serve to you at your request. Why not?
It's a community project sponsored by donations.
[1] Eyeballing the trend on this page - https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia which mentions "As of May 2015, the current version of the English Wikipedia article / template / redirect text was about 51 GB uncompressed in XML format", so I estimate based on the shape of the graph that it is less than 100 GB today.
- taejo 10y agoYou can download a database dump which contains the wikitext source. You can interpret this into HTML. What GP means is extracting the natural-language text from this (separating it from the infoboxes, presentation information, etc).
- logicallee 10y agooh, okay. thanks.