5 ms·
TIL entire wikipedia is less than 100 GB when compressed.
by leowoo91 8y ago
TIL entire wikipedia is less than 100 GB when compressed.
- arthurcolle 8y agoThat's with no images, and iirc it also didn't include any of the math markup last time I looked at one of the xml dumps in one of the loaders (think it was kiwix)
- leowoo91 8y agoTotally forgot about images. I just dug little further and found this: "The size of the media files in Wikimedia Commons, which includes the images, videos and other media used across all the language-specific Wikipedias was described as well over 23 TB near the end of 2014" Considering 30 TB hdd costs around 1500$, still interesting.
- crtasm 8y agoIf you grab those WD Easystore 8tb drives when they're on sale you can have 32tb for around $560
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- arthurcolle 8y agoI wonder what the size is as of this moment, maybe double? I read somewhere recently that 90% of the content on the Web was created in the last 2 years. Not sure if thats "special media" focused, i.e. video & pictures, as opposed to text. But I'm sure the size of the dump has increased substantially in the last 4 years.
- leowoo91 8y agoVery likely, there at least need of a filter at some level to size down (maybe getting single language or converting videos to low-res)
- kopo 8y agoYou can find versions that are 50GB if you dig around the kiwix download page! The entire stackoverflow dump is around the same size.