3 ms·
That's with no images, and iirc it also didn't include any of the math markup last time I looked at one of the xml dumps in one of the loaders (think it was kiw
by arthurcolle 8y ago
That's with no images, and iirc it also didn't include any of the math markup last time I looked at one of the xml dumps in one of the loaders (think it was kiwix)
- leowoo91 8y agoTotally forgot about images. I just dug little further and found this: "The size of the media files in Wikimedia Commons, which includes the images, videos and other media used across all the language-specific Wikipedias was described as well over 23 TB near the end of 2014" Considering 30 TB hdd costs around 1500$, still interesting.
- crtasm 8y agoIf you grab those WD Easystore 8tb drives when they're on sale you can have 32tb for around $560
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- arthurcolle 8y agoI wonder what the size is as of this moment, maybe double? I read somewhere recently that 90% of the content on the Web was created in the last 2 years. Not sure if thats "special media" focused, i.e. video & pictures, as opposed to text. But I'm sure the size of the dump has increased substantially in the last 4 years.
- leowoo91 8y agoVery likely, there at least need of a filter at some level to size down (maybe getting single language or converting videos to low-res)