3 ms·
That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head. A
by giancarlostoro 9y ago
That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.
As for anyone curious:
https://dumps.wikimedia.org/backup-index.html https://dumps.wikimedia.org/backup-index.html
- 3131s 9y agoThe most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here: https://dumps.wikimedia.org/enwiki/20180320/ https://dumps.wikimedia.org/enwiki/20180320/ The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.
- lacksconfidence 9y agoYou can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata) http://dumps.wikimedia.your.org/other/cirrussearch/ http://dumps.wikimedia.your.org/other/cirrussearch/ These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump