3 ms·

7TB of compressed text? I don't think humanity has generated that much written words in it's entire existence. Although it would be an interesting Fermi Problem

by ars 4y ago

7TB of compressed text? I don't think humanity has generated that much written words in it's entire existence. Although it would be an interesting Fermi Problem to estimate (and don't forget just how well text compresses). This has to be a lot of duplicates or bad formats (images). This would be far more useful to people with some curating.

khazhoux 4y ago
These are most likely PDFs, not plaintext.
deleted 4y ago
[deleted]
Aachen 4y ago
(Ignoring, as someone else already pointed out, that this is PDFs with images and scans probably, not just plain text... I got curious and did some napkin math about your claim) The top ten countries average north of a hundred thousand books per year each[1], so let's say a million books per year globally because I haven't got the time to extract the numbers and sum them all up exactly. It probably wasn't as much fifty years ago, but we're also completely ignoring the Internet which is way bigger because of user-contributed content (not to mention things like newspapers, meeting notes, etc.), so let's say this was the case for the past fifty years. That's fifty million books. Average book has 85k words[2], and a word is like 5 characters on average. Not all countries write in English, especially some big ones like India and China, so we could probably double it on a global scale but let's go for a conservative 7.5 bytes per word and add a byte for the space (I'm ignoring other punctuation). That comes out to 8.5×85e3×1e6 which is less than a gigabyte and uncompressed. Decent compression iirc makes it a fifth of the size, so 145MB compressed. I've still got to be an order of magnitude off because we write a whole lot more than just books (and even if we were just looking at books: there are also book revisions, drafts, etc. I'm just counting the published words), but that's still less than expected. In conclusion, it comes out to about 1/50'000th of 7TB compressed, which I would say lends some credence to the claim—even if I feel like I must have made a mistake somewhere because 145MB for ~all books of the past 50 years from all countries seems quite little. [1] https://en.wikipedia.org/wiki/Books_published_per_country_per_year https://en.wikipedia.org/wiki/Books_published_per_country_pe... [2] a few sources on ddg, e.g. https://www.tckpublishing.com/how-long-should-a-book-be/ https://www.tckpublishing.com/how-long-should-a-book-be/
deleted 4y ago
[deleted]

by ars 4y ago

khazhoux 4y ago

These are most likely PDFs, not plaintext.

deleted 4y ago

[deleted]

Aachen 4y ago

(Ignoring, as someone else already pointed out, that this is PDFs with images and scans probably, not just plain text... I got curious and did some napkin math about your claim) The top ten countries average north of a hundred thousand books per year each[1], so let's say a million books per year globally because I haven't got the time to extract the numbers and sum them all up exactly. It probably wasn't as much fifty years ago, but we're also completely ignoring the Internet which is way bigger because of user-contributed content (not to mention things like newspapers, meeting notes, etc.), so let's say this was the case for the past fifty years. That's fifty million books. Average book has 85k words[2], and a word is like 5 characters on average. Not all countries write in English, especially some big ones like India and China, so we could probably double it on a global scale but let's go for a conservative 7.5 bytes per word and add a byte for the space (I'm ignoring other punctuation). That comes out to 8.5×85e3×1e6 which is less than a gigabyte and uncompressed. Decent compression iirc makes it a fifth of the size, so 145MB compressed. I've still got to be an order of magnitude off because we write a whole lot more than just books (and even if we were just looking at books: there are also book revisions, drafts, etc. I'm just counting the published words), but that's still less than expected. In conclusion, it comes out to about 1/50'000th of 7TB compressed, which I would say lends some credence to the claim—even if I feel like I must have made a mistake somewhere because 145MB for ~all books of the past 50 years from all countries seems quite little. [1] https://en.wikipedia.org/wiki/Books_published_per_country_per_year https://en.wikipedia.org/wiki/Books_published_per_country_pe... [2] a few sources on ddg, e.g. https://www.tckpublishing.com/how-long-should-a-book-be/ https://www.tckpublishing.com/how-long-should-a-book-be/

deleted 4y ago

[deleted]