6 ms·

How much space would it take to store every word ever said?

49 points by jonluca 7y ago

jmull 7y ago
Hm... just the words loses so much -- the tone, the emphasis, the pauses. I think we'd have to do at least audio. Though of course expressions, hand movements and bearing count too, so I'm thinking we need a number for video as well.
- javajosh 7y ago
  That's only a concern if you're want to store some measure of identity and/or meaning; the author just wants to store words. In any case, you could improve things by putting words into a screenplay-like format, which implies relatively small amount of additional text. (I would estimate a typical screenplay or play script is 90% dialogue).
deleted 7y ago
[deleted]
perl4ever 7y ago
Assuming 10:1 compression, you have 50 exabytes, and it appears that would be about 500 of the trucks Amazon uses to load large amounts of data. I can't find information on how many they actually have, or whether the capacity has increased from the 100 PB figure mentioned in a lot of places. Amazon's FAQ is funny: "Q: Can I export data from AWS with Snowmobile? Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS" ...you can check out any time you like, but you can never leave.
- otabdeveloper2 7y ago
  It's not funny, it's sad. Also, AWS sucks. It's the 21st century version of IBM mainframe, except without the reliability.
- tetris11 7y ago
  Such a lovely place
Avshalom 7y ago
It's not taking into account UTF-8 though so maybe double or triple.
- saagarjha 7y ago
  You could probably do this with minimal overhead by organizing the words by language assuming a codepage for each set.
- willis936 7y ago
  Encoding schemes like UTF-8 don’t affect compressed size much. What matters is the quantity of information.
DoofusOfDeath 7y ago
Sometimes I hear someone utter a sentence which I guess has never before been uttered by anybody. I really wish I had a way to verify that, just for fun.
- farnsworth 7y ago
  You could check whether it's already been posted to /r/BrandNewSentence/
  - oarabbus_ 7y ago
    That must be an absurdly small subset of the universe of sentences; for example, this sentence itself has likely never been spoken or typed until now.
    - clSTophEjUdRanu 7y ago
      There are 7.06e57 combinations of playing cards. Odds are that every time someone shuffles a deck of cards it's a unique shuffling.
    - tetris11 7y ago
      I found your exact comment in the library of babel, page 82 of a book. https://libraryofbabel.info/bookmark.cgi?m_hnrvffbgsqubi.82 https://libraryofbabel.info/bookmark.cgi?m_hnrvffbgsqubi.82
- psychoslave 7y ago
  I came with "Unique first utterer of this performative impredicative statement". I forged it as a profile title[1], and so far no one dared to contradict me. :P Note that I won't lose my time in a preterition which is giving credits to Russell for the concept impredicativity and to Austin for the concept of performative sentence. [1] https://www.linkedin.com/in/mathieu-lovato-stumpf-guntz-%E2%AD%90-83413884/ https://www.linkedin.com/in/mathieu-lovato-stumpf-guntz-%E2%...
- bananasbandanas 7y ago
lilyball 7y ago
> 10 billion words, times an average word length of 11.66, gets us ~4.8 billion individual characters spoken per person per lifetime. Am I missing something or is this math very wrong?
- ringshall 7y ago
  Also, “ roughly 10 billion words said, per person (16,000 per day * 26,280 days = ~420,480,000)“ Not sure if I completely missed something but it looks like 420m words per person. Edit: combine these two errors and it makes sense. The 10b number is in the text but not the arithmetic. 16000 words per day * 26000 days * 11 chars per word = 4.5E9 chars per person.
- isomorphic 7y ago
  I noticed this, too. If you take 16,000 words/day multiplied by 26,280 days (72 years' worth of days), you get the 420.48 million in the text. If you take that number and multiply by 24, you get 10,091,520,000--close enough to 10 billion that I think the author made a days versus hours mistake somewhere. The remainder of the article seems to actually use the 420.48 million number and not 10 billion, as ringshall points out.
- jonluca 7y ago
  That's my bad, accidentally left in some earlier wording. All the numbers are correct except for references to 10 billion - those should all be 420 million. It should be updated in the text soon.
KiDD 7y ago
But what if it was encoded in a novel storage format such as DNA?
- willis936 7y ago
  How can DNA be a novel storage format when its use for information storage predates computers by at least 3 billion years?
  - duxup 7y ago
    Well novel for us to use it in this way.... Hummm, better patent "do everyday things...in DNA"
franciscop 7y ago
> We could also use UTF8, but since we assumed the language is German, we’ll stick to ASCII German cannot be expressed in ASCII[1]. For that fact, neither can Chinese nor Spanish, the two most spoken languages besides English. Also UTF8 doesn't even encode all the languages ever spoken. So IMHO this is at least an order of magnitude wrong. [1] https://news.ycombinator.com/item?id=9222071 https://news.ycombinator.com/item?id=9222071
- deleted 7y ago
  [deleted]
- someperson 7y ago
  They probably meant ISO/IEC 8859-15 8-bit encoding https://en.wikipedia.org/wiki/ISO/IEC_8859-15 https://en.wikipedia.org/wiki/ISO/IEC_8859-15
- QuadrupleA 7y ago
  Each character in e.g. Chinese represents more information, so there are fewer of them, which sorta cancels out. I thought German was a good conservative choice here.
- dspig 7y ago
  German can be expressed on ascii just fine - ae oe ue ss are commonly used and understood when the non-ascii characters are unavailable or just hard to find. So for this purpose would be fine, except possibly introducing ambiguity in some words or names that already use the ascii version.
- jl6 7y ago
  For what it’s worth, there are English words which cannot be encoded in ASCII too. https://en.m.wikipedia.org/wiki/English_terms_with_diacritical_marks https://en.m.wikipedia.org/wiki/English_terms_with_diacritic...
- deleted 7y ago
  [deleted]
purplezooey 7y ago
You can leave out all of Twitter
QuadrupleA 7y ago
This would be an interesting dataset to explore! A biographer's dream. Insider information on every corporate & governmental decision in history. Intimate daily-life details from early hominids.
sudosushi 7y ago
Interestingly, no one has mentioned the Library of Babel[0] One could assert that if you were to translate Chinese/Russian/non-UTF characters into UTF, you'd be covering every word ever possibly said. [0] https://libraryofbabel.info/ https://libraryofbabel.info/
thansz 7y ago
This reminds me of a very fun and interesting read called "A Short Stay in Hell" by Steven Peck, which provides an entertaining perspective on infinity and very, very large finite time periods. It's about a Mormon who goes to hell (because Zoroastrianism happens to be the One True Religion). Hell does not last forever though. For the main character, it's a library that contains every possible communication that could exist. Once he finds the book that contains the story of his life, he gets out. Very fun read that addresses large but finite values, although it focuses more on time rather than space.
slewis 7y ago
If we can assume determinism, there’s a much better compression algorithm.

49 points by jonluca 7y ago

Hm... just the words loses so much -- the tone, the emphasis, the pauses. I think we'd have to do at least audio. Though of course expressions, hand movements and bearing count too, so I'm thinking we need a number for video as well.

javajosh 7y ago
That's only a concern if you're want to store some measure of identity and/or meaning; the author just wants to store words. In any case, you could improve things by putting words into a screenplay-like format, which implies relatively small amount of additional text. (I would estimate a typical screenplay or play script is 90% dialogue).

deleted 7y ago

[deleted]

perl4ever 7y ago

Assuming 10:1 compression, you have 50 exabytes, and it appears that would be about 500 of the trucks Amazon uses to load large amounts of data. I can't find information on how many they actually have, or whether the capacity has increased from the 100 PB figure mentioned in a lot of places. Amazon's FAQ is funny: "Q: Can I export data from AWS with Snowmobile? Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS" ...you can check out any time you like, but you can never leave.

otabdeveloper2 7y ago
It's not funny, it's sad. Also, AWS sucks. It's the 21st century version of IBM mainframe, except without the reliability.
tetris11 7y ago
Such a lovely place

Avshalom 7y ago

It's not taking into account UTF-8 though so maybe double or triple.

saagarjha 7y ago
You could probably do this with minimal overhead by organizing the words by language assuming a codepage for each set.
willis936 7y ago
Encoding schemes like UTF-8 don’t affect compressed size much. What matters is the quantity of information.

DoofusOfDeath 7y ago

Sometimes I hear someone utter a sentence which I guess has never before been uttered by anybody. I really wish I had a way to verify that, just for fun.

farnsworth 7y ago
You could check whether it's already been posted to /r/BrandNewSentence/
- oarabbus_ 7y ago
  That must be an absurdly small subset of the universe of sentences; for example, this sentence itself has likely never been spoken or typed until now.
  - clSTophEjUdRanu 7y ago
    There are 7.06e57 combinations of playing cards. Odds are that every time someone shuffles a deck of cards it's a unique shuffling.
  - tetris11 7y ago
    I found your exact comment in the library of babel, page 82 of a book. https://libraryofbabel.info/bookmark.cgi?m_hnrvffbgsqubi.82 https://libraryofbabel.info/bookmark.cgi?m_hnrvffbgsqubi.82
psychoslave 7y ago
I came with "Unique first utterer of this performative impredicative statement". I forged it as a profile title[1], and so far no one dared to contradict me. :P Note that I won't lose my time in a preterition which is giving credits to Russell for the concept impredicativity and to Austin for the concept of performative sentence. [1] https://www.linkedin.com/in/mathieu-lovato-stumpf-guntz-%E2%AD%90-83413884/ https://www.linkedin.com/in/mathieu-lovato-stumpf-guntz-%E2%...
bananasbandanas 7y ago

lilyball 7y ago

> 10 billion words, times an average word length of 11.66, gets us ~4.8 billion individual characters spoken per person per lifetime. Am I missing something or is this math very wrong?

ringshall 7y ago
Also, “ roughly 10 billion words said, per person (16,000 per day * 26,280 days = ~420,480,000)“ Not sure if I completely missed something but it looks like 420m words per person. Edit: combine these two errors and it makes sense. The 10b number is in the text but not the arithmetic. 16000 words per day * 26000 days * 11 chars per word = 4.5E9 chars per person.
isomorphic 7y ago
I noticed this, too. If you take 16,000 words/day multiplied by 26,280 days (72 years' worth of days), you get the 420.48 million in the text. If you take that number and multiply by 24, you get 10,091,520,000--close enough to 10 billion that I think the author made a days versus hours mistake somewhere. The remainder of the article seems to actually use the 420.48 million number and not 10 billion, as ringshall points out.
jonluca 7y ago
That's my bad, accidentally left in some earlier wording. All the numbers are correct except for references to 10 billion - those should all be 420 million. It should be updated in the text soon.

KiDD 7y ago

But what if it was encoded in a novel storage format such as DNA?

willis936 7y ago
How can DNA be a novel storage format when its use for information storage predates computers by at least 3 billion years?
- duxup 7y ago
  Well novel for us to use it in this way.... Hummm, better patent "do everyday things...in DNA"

franciscop 7y ago

> We could also use UTF8, but since we assumed the language is German, we’ll stick to ASCII German cannot be expressed in ASCII[1]. For that fact, neither can Chinese nor Spanish, the two most spoken languages besides English. Also UTF8 doesn't even encode all the languages ever spoken. So IMHO this is at least an order of magnitude wrong. [1] https://news.ycombinator.com/item?id=9222071 https://news.ycombinator.com/item?id=9222071

deleted 7y ago
[deleted]
someperson 7y ago
They probably meant ISO/IEC 8859-15 8-bit encoding https://en.wikipedia.org/wiki/ISO/IEC_8859-15 https://en.wikipedia.org/wiki/ISO/IEC_8859-15
QuadrupleA 7y ago
Each character in e.g. Chinese represents more information, so there are fewer of them, which sorta cancels out. I thought German was a good conservative choice here.
dspig 7y ago
German can be expressed on ascii just fine - ae oe ue ss are commonly used and understood when the non-ascii characters are unavailable or just hard to find. So for this purpose would be fine, except possibly introducing ambiguity in some words or names that already use the ascii version.
jl6 7y ago
For what it’s worth, there are English words which cannot be encoded in ASCII too. https://en.m.wikipedia.org/wiki/English_terms_with_diacritical_marks https://en.m.wikipedia.org/wiki/English_terms_with_diacritic...
deleted 7y ago
[deleted]

purplezooey 7y ago

You can leave out all of Twitter

QuadrupleA 7y ago

This would be an interesting dataset to explore! A biographer's dream. Insider information on every corporate & governmental decision in history. Intimate daily-life details from early hominids.

sudosushi 7y ago

Interestingly, no one has mentioned the Library of Babel[0] One could assert that if you were to translate Chinese/Russian/non-UTF characters into UTF, you'd be covering every word ever possibly said. [0] https://libraryofbabel.info/ https://libraryofbabel.info/

thansz 7y ago

This reminds me of a very fun and interesting read called "A Short Stay in Hell" by Steven Peck, which provides an entertaining perspective on infinity and very, very large finite time periods. It's about a Mormon who goes to hell (because Zoroastrianism happens to be the One True Religion). Hell does not last forever though. For the main character, it's a library that contains every possible communication that could exist. Once he finds the book that contains the story of his life, he gets out. Very fun read that addresses large but finite values, although it focuses more on time rather than space.

slewis 7y ago

If we can assume determinism, there’s a much better compression algorithm.