4 ms·
I was about to say that a torrent is hardly necessary. How big could a 1000 mostly-text files get? Pretty big as it turns out. Downloading a dozen random entrie
by computator 7y ago
I was about to say that a torrent is hardly necessary. How big could a 1000 mostly-text files get? Pretty big as it turns out. Downloading a dozen random entries from that list, the sizes seem average around 2 MB, and that's including only the small images on each page (not the big picture you get when you click on an image). So 1000 entries at 2 MB each would be 2 GB.
Picking apart just one page (the Jane Austen entry), the plain ASCII text with no markup is only 88 KB. The 19 small images, plus some tiny buttons and logos, are 536 KB, and the markup (HTML, CSS, and whatnot) is 497 KB. I was surprised that Wikipedia, in terms of page weight, is mostly images and markup. (Not complaining, of course. Wikipedia is one of the few big sites on the web that doesn't throw in gratuitous and irrelevant images and videos.)
- DanBC 7y agoAre you including page history? Are you allowed to re-distribute without page history? Also, you probably know this but there is a prize for compressing bits of Wikipedia. Here's the wikipedia page about it: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize http://prize.hutter1.net/ http://prize.hutter1.net/ About the test data: http://mattmahoney.net/dc/textdata.html http://mattmahoney.net/dc/textdata.html And discussion on HN: https://news.ycombinator.com/item?id=7405129 https://news.ycombinator.com/item?id=7405129