3 ms·
To be fair wikipedia compressed without images is pretty small. A couple tens of gigs no more than 80. You can download it to your phone and there are apps to v
by giancarlostoro 9y ago
To be fair wikipedia compressed without images is pretty small. A couple tens of gigs no more than 80. You can download it to your phone and there are apps to view it all. Not sure how larger it is decompressed but 2 TB sounds like overkill. But yeah wikipedia would be a good way to test it.
Also did you mean every single wikipedia site or just the English one? I'm thinking just the English one's sufficient.
- tomcooks 9y agoI know what you mean with "sufficient",but English pages are often lacking informations on a lot of subjects related to countries outside the anglosphere. I still don't understand why wikipedia articles aren't translated according to a common script, at least where there are featured translations (marked a star)
- zeckalpha 9y agoFeel free to make that happen.
- WalterGR 9y agoAlso, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?
- giancarlostoro 9y agoThat would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head. As for anyone curious: https://dumps.wikimedia.org/backup-index.html https://dumps.wikimedia.org/backup-index.html
- 3131s 9y agoThe most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here: https://dumps.wikimedia.org/enwiki/20180320/ https://dumps.wikimedia.org/enwiki/20180320/ The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.
- lacksconfidence 9y agoYou can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata) http://dumps.wikimedia.your.org/other/cirrussearch/ http://dumps.wikimedia.your.org/other/cirrussearch/ These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump