8 ms·
Researchers: It takes 1.5 MB of data to store language information
- admax88q 8y ago1.5 million what? Fix your title.
- deleted 8y ago[deleted]
- LKAndrew 8y agoHow would it be possible that 40,000 words translates to 400,000 bits?! Am I missing something here?
- whitten 8y agoMy only guess is that they are implicitly ignoring information we use all the time when talking about information in a computer. The number 5 stored in one place in a computer is the same as the number 5 stored elsewhere. In the human brain two words are already different because they are in different places. The researchers aren’t counting information needed to tell them apart. Admittedly, this is not the way I think either.
- simias 8y agoCompression I suppose. Like storing the word "compression" is easier if you already know the familiar "com-" prefix as well as the noun "pression". Which is itself easier to remember if you know the verb "to press". I've been actively learning Portuguese and Russian lately, it's impressive how much faster I can pick up Portuguese vocabulary vs. Russian. And that's even for words that don't have an obvious cognate in languages I already know. The structure of the words, the various building blocks are just so much more familiar in Portuguese. A word like "atrever" (to dare) doesn't have any obvious cognate in languages that I know but it just "looks right" it a way that, say, "atverer" or "aterver" wouldn't. Those last two words sound distinctly un-Portuguese (I might say, un-Romance) to me. That makes it a lot easier to remember the spelling. Eventually as I grow my Russian vocabulary I start making similar connections. Волноваться is pretty tricky to memorize on its own but it becomes easier when you know that ~ся is the reflective, ~ть is the common verb ending, волна means wave and ~овать is a very common building block for Russian verbs.
- mrec 8y ago"Compression" applies very much to grammatical variants of words. It's why the only bizarre irregular verbs in a language tend to be the ones that get used all the time - be, do, go etc - because for anything more obscure the brain just forgets the special case and applies the general rule. Steven Pinker's book Words and Rules is a great layman-oriented read if this sort of thing interests you.
- coldtea 8y agoIt also makes sense why the most common verbs are irregular in most languages: to have us pick the direct word quickly, instead of the slower way of deriving it from a rule. So, they're like "constants" vs calling a function to calculate a value.
- mrec 8y agoNot sure I follow. If (big if) there were really a measureable advantage to having us "pick the direct word quickly, instead of the slower way of deriving it from a rule", it doesn't follow that irregularity makes that easier. I could memorize "goed" just as easily as I can memorize "went".
- simias 8y agoI took it the other way around: it's easy to memorize "went" because you use it all the time. If on the other hand a much less common verb like "to satiate" had a very irregular conjugation then it would regularize pretty quickly because nobody but ultra-pedants would bother to remember the exception. I think a decent real example of that is fiancé/fiancée, those are french borrowings and have, at least originally, kept the French grammatical gender inflection. However nowadays I often see people using either spelling in a gender-neutral way since most people don't bother to learn French grammar for this one word.
- coldtea 8y ago>I took it the other way around: it's easy to memorize "went" because you use it all the time. That still wouldn't explain the why of having it like "went" vs "goed". Sure, it's easy to memorize because we use it all the time, but why have it to memorize it in the first place, versus something like "goed". So, this theory (I tried to convey above) said that it being irregular placed ensured we don't slow down try to derive it from regular rules, but instead have fast access to a memorized form. Couldn't we just memorize "goed"? If it's just "frequency of use" that mattered, "went" and "goed" would work just as well. But the extra idea is that "goed", being regular, would be too easy for us to confuse with thousand of other regular verbs, and not use our "fast recall" mechanism, regardless of that verb being needed all the time. Not sure if correct - read it years ago. This seems to be related to that: https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Linguistic_study https://en.wikipedia.org/wiki/Regular_and_irregular_verbs#Li...
- shanth 8y agoSuperposition / quantum mechanical
- GuB-42 8y agoI've seen somewhere that there is about 1 bit of entropy per letter of English text. The best packer (Hutter prize) compresses 100MB of wikipedia down to 15MB, including the packer itself, a ratio of 1:6.5, which isn't that far off. Considering that, 40k words to 400kbits is not too surprising.
- jerf 8y agoThe number of "bits" something takes is not an absolute value. It is relative to the encoding scheme that is being used for the bits. Converting a large word list to a small number of bits has been a computer science hobby for a long time. Here's a pretty good search result to start working through for more details: https://duckduckgo.com/?q=building+a+small+spell+checker+suffix+tree https://duckduckgo.com/?q=building+a+small+spell+checker+suf... It was especially important to write small and fast spell checkers in the 1980s and early 1990s, when you couldn't expect to have enough RAM sitting around to simply load up a naively-encoded list of words, and the act of spell checking a few thousand words could take noticeable time. So in an encoding scheme chosen to represent English compactly, I'm not too surprised that you can get things down quite small. However, the question is, what relevance does that encoding scheme have to the human brain? Having just scanned through the paper, the answer is "probably not that much", which the researchers are well aware of. They explicitly present this as a lower-bound, which is a reasonable thing to do. It is obvious that the brain does not simply store 1.5MB of data in the way a computer does, in many ways. To be honest, this amounts to an exercise in recreational mathematics more than anything else. There's nothing wrong with that, and that's not a criticism. My point is that I'm not sure it's worth trying to read the paper as anything else.
- boomlinde 8y agoA quick test using a non-lossy compressor with no understanding of phonemes or human language and grammatical context at all on a dictionary of 370000 English words resulted in 24 bits per word here. It wouldn't surprise me if our abilities to roughly contextualize language in terms of the language we already understand gives us a serious advantage here. Now a few questions: Can you hear a word you've never written (in a language that you're familiar with) and intuitively spell it right the first time? Can you read a word that you don't immediately understand and figure out its meaning from the context in which it is used? Can you accurately complete half of a sentence? That a lot of people can do these things suggests to me that we all sit on something superficially similar to an efficient lossy compressor in our brain.
- oeuviz 8y agoSo, all of it fits on a floppy? Nice.
- timw4mail 8y agoNot quite. Considering the most common floppies stored a bit less than 1.44MB.
- segfaultbuserr 8y agoOne feature of floppy disks is that the recording technology and the format is independent from the physical medium, the limitations are mostly imposed by the disk drive, the encoding and the file system, not the medium itself. Well, this "feature" creates significant compatibility issues as all the vendors created their own proprietary formats, sometimes with patents. As a result, together with the economics of scale, often, the standard format is not the best design, a simpler design that serves as the lowest common denominator. On the other hand, due to this feature, it was already possible to store 1,760 KB of data on a 3-1/2 inch HD floppy on a 1986 Amiga. And in 2000, it was possible to store 32 MiB of data on a standard 1.44 MB floppy disk, by using a SuperDisk LS-240 drive (although random write is sacrificed, the entire floppy must be rewritten if a change is needed, like CD-RWs). I believe the advancements in magnetic recording technology in the past 20 years allows one to achieve even higher capacity on a standard floppy, and it can be as cheap as early floppy drives if mass produced, only that it doesn't make sense to do so.
- MisterTea 8y agoI always wanted to know how far you could push the capacity of an old standard density floppy drive if one hacked on modern control electronics. Things like adding more tracks and variable sectors per track. Though I think you'd be limited by the magnetic density of the media and physical head size. Still, it would be fun to see just how much more data you could squeeze onto an old 1.44MB floppy.
- timw4mail 8y ago
- messe 8y agoMakes me want to write a retrocyberpunk setting featuring languages loaded off floppy disks.
- segfaultbuserr 8y agoDo it! Preferably, the floppy disks should be the 5-1/4 inch 1.2 MiB high-density, double-sided type, 3-1/2 inch disks sounds good too ;-)
- dmos62 8y agoDo it!
- MrQuincle 8y agoThis describes how to summarize a language in a computer system. I think the storage requirements for the semantics is in particular questionable. If you can refer to a particular concept somewhere in the brain you only need a pointer to it. However, single pointers might not be sufficient The apparatus to learn a language might also not so easily distinguished from the structure to store the language or the structure to quickly recall the language. In that case taking about storage is like talking about compressing Plato's world of concepts.
- coldtea 8y ago>This describes how to summarize a language in a computer system. It describes how to summarize a language from a information theory standpoint (which is something different). There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation (and even if not, it's a good lower bound)
- madhadron 8y ago> There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation Do you have a citation? I would be very interested to see it. I hate to make analogies between computing and biological systems, but I immediately think of denormalization in databases, error correcting codes, and database replication as examples of optimizations that take you farther from information theoretic efficiency. Information applies very straightforwardly to the study of a spike train of a synapse (source, channel, and receiver). I remain unconvinced that it systems like memory make sense as information channels.
- pmayrgundter 8y agoShannon observed this in his work towards information theory. Humans are able to predict next letters in a stream of natural language with a probability approaching optimal. The original paper is here: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf The summary at wikipedia: https://en.wikipedia.org/wiki/Entropy_(information_theory)#Entropy_as_information_content https://en.wikipedia.org/wiki/Entropy_(information_theory)#E... Notably, PPM, the best compression algorithm (at the time), was not able to predict as well. So that's a quantitative demonstration of optimality in language performance that includes a faculty for understanding syntax and semantics and presumably is linked to phonology, pragmatics etc. Chomsky discusses optimality in a 1998 talk here: https://youtu.be/7Sw15-vSY8E https://youtu.be/7Sw15-vSY8E, at about 1h 7m an onwards. He says language evolved rather quickly for such a significant function (estimating it arose quickly ~200kya). He says though many things in evolution are pretty messy, language is close to optimal design. He doesn't provide references, but I think (given the time and academic associations; anyone know?) he may be referring to Optimality Theory: https://en.wikipedia.org/wiki/Optimality_Theory https://en.wikipedia.org/wiki/Optimality_Theory Separately, our knowledge of anatomy of the language faculty is that it is highly localized in the Wernicke and Broca areas of the brain. This also suggests a small modification of an existing brain structure. Additionally, optimality in biological systems has been a productive research direction in a few areas: - Optimality Principles in Biology - https://link.springer.com/book/10.1007/978-1-4899-6419-9 https://link.springer.com/book/10.1007/978-1-4899-6419-9 - Near optimal energy use in the metabolism of cell division - https://arxiv.org/pdf/1209.1179.pdf https://arxiv.org/pdf/1209.1179.pdf - https://www.quantamagazine.org/the-math-that-tells-cells-what-they-are-20190313/ https://www.quantamagazine.org/the-math-that-tells-cells-wha...
- wrinkl3 8y agoI haven't read the paper itself, but the article makes it sound like they simply counted the minimum number of bits required to represent the English language. It mentions no neuroscience-specific insights, so the "brain" part of the title is quite misleading.
- rob74 8y agoAside from that - is there really a way to express neural connections in the brain in terms of bits and bytes?
- jerf 8y agoThe problem isn't whether there is "a" way; the problem is there's a multiplicity of ways and we don't know what the best one is. Part of the reason we don't have any idea what the best is is that we can't even construct any of them from a real brain, that is, the "brain scanning" technology the techno-rapture-style Singularity expects to develop does not even remotely exist right now. So we have nothing to experiment with or gain any experience with. The only ones we'd even remotely have access to right now would be some rather brute-force encoding of neurons and connectivity strengths and we pretty much know that such a thing must be incredibly redundant, but we have no ability right now to remove that redundancy. And we can't do it on a very large neural set. It has been done on small models, though, at least to some significant extent: https://en.wikipedia.org/wiki/OpenWorm https://en.wikipedia.org/wiki/OpenWorm
- aperrien 8y agoBrain scanning techniques and simulation software have a long way to go, but there have been significant advances made: A full fly connectome: https://www.nature.com/articles/s41684-018-0183-8 https://www.nature.com/articles/s41684-018-0183-8 Architecture of the Mouse Brain Synaptome: https://www.cell.com/neuron/fulltext/S0896-6273(18)30581-6 https://www.cell.com/neuron/fulltext/S0896-6273(18)30581-6 So we are past just a simple worm simulation now.
- 8y ago
- werber 8y agoThis seems very low. Just thinking of all of the different ways you can greet someone. The inflection and modifiers based on circumstance and audience alone seems like it would be higher than 1.5mb
- adrianN 8y agoI believe it if they provide a program that can speak English in under 1.5MB total size.
- tobyhinloopen 8y agoWhat does it mean to "speak English"? Does a program that plays a recording of the word "I" count as a program that speaks English?
- adrianN 8y agoI think you know the answer: think about how you find out whether a human speaks English or not.
- cs2818 8y agoI haven't had a chance to look at it yet but here are the links to the paper and associated Open Science Framework data if anyone is interested: Paper: https://royalsocietypublishing.org/doi/full/10.1098/rsos.181393 https://royalsocietypublishing.org/doi/full/10.1098/rsos.181... OSF Page: https://osf.io/ga9th/ https://osf.io/ga9th/
- mannykannot 8y agoOf the various estimates that go into this number, I am most skeptical of the semantics one. I doubt that one can draw a clear line between our semantic appreciation of language and the totality of our remembered experiences.
- wmnwmn 8y agoHaha wrong