6 ms·

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-boosts-the-human-brain-s-memory-capacity-10-fold/ https://www.scientificamerican.com/article/new-estimate-boos...

downboots 3y ago
Can "distill knowledge" be made precise ?
- ComputerGuru 3y ago
  As best as I’ve been able to research, it’s still under active exploration and there are hypotheses but no real answers. I believe research has basically been circling around the recent understanding that in addition to being part of how the brain is wired, it is also an active, deliberate (if unconscious) mechanism that takes place in the background and is run “at a higher priority” during sleep (sort of like an indexing daemon running at low priority during waking hours then getting the bulk of system resources devoted to it during idle). There are also studies that show “data” in the brain isn’t stored read-only and the process of accessing that memory involves remapping the neurons (which is how fake memories are possible) - so my take is if you access a memory or datum sequentially start to finish each time the brain knows this is to be stored verbatim for as-is retrieval but if you access snapshots of it or actively seek to and replay a certain part while trying to relate that memory to a process or a new task, the brain rewires the neural pathways accusingly. Which implies that there us an unconscious part that takes place globally plus an active, modifying process where how we use a stored memory affects how it is stored and indexed (so data isn’t accessed by simple fields but rather by complex properties or getters, in programming parlance). I guess the key difference from how machine learning works (and I believe an integral part of AGI, if it is even possible) is that inference is constant, even when you’re only “looking up” data and you don’t know the right answer (i.e. not training stage). The brain recognizes how the new query differs from queries it has been trained on and can modify its own records to take into account the new data. For example, let’s say you’re trying to classify animals into groups and you’ve “been trained” on a dataset that doesn’t include monotremes or marsupials. The first time you come across a platypus in the wild (with its mammaries but no nipples, warm-blooded but lays eggs, and a single duct for waste and reproduction) you wouldn’t just mistakenly classify it as a bird or mammal - you would actively trigger a (delayed/background) reclassification of all your existing inferences to account for this new phenomenon, even though you don’t know what the answer to the platypus classification question is.
- clord 3y ago
  imo, it amounts to revisiting concepts once more general principles are found — and needed. For instance, you learn the alphabet, and it's hard. the order is tricky. the sounds are tricky, etc. but eventually, it get distilled to a pattern. But you still have to start from A to remember what letter 6 is, until you encounter that problem many times, and then the brain creates a 6=F mapping. I think of it in economic terms: when the brain realizes it's cheaper to create a generalization, it does so on the fly, and that generalization takes over the task. Somtimes it's almost like creating a specialist shard to take over the task. Driving is hard at first, with very high task overload, lots to pay attention to. With practice, it becomes a little automated part of yourself takes care of those tasks while your main general intelligence can do whatever it likes, even as the "driver" deals with seriously difficult tasks.
  - pyinstallwoes 3y ago
    It’s almost like a tuning fork. When the mapping becomes 6=F the two separate tuning forms are now sympathetic to a frequency.
- esafak 3y ago
  https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory
- __loam 3y ago
  Unless you know something the neuroscientists don't, it cannot.
jjk166 3y ago
Distilling knowledge is data compression.
- w10-1 3y ago
  You're conflating memorization with generalization, no?
  - jjk166 3y ago
    Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.
    - staunton 3y ago
      > Generalization is developing the heuristics by which you compress stored data such that > you nevertheless can recover enough information to be useful in the future. I disagree (in case you meant to imply it) that compression implies generalization.
      - jjk166 3y ago
        I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.
nonameiguess 3y ago
I've thought about this a lot in the context of the desire people seem to have to try and achieve human immortality or at least indefinite lifespans. If SciAm is correct here and the upper bound is a quadrillion bytes, we may not be able to hit that given the bound on possible human experiences, but someone who lived long enough would eventually hit that. After a hundred million years or whatever the real number is of life, you'd either lose the ability to form new memories or you'd have to overwrite old ones to do so. Aside from having to eventually experience the death of all stars and light and the decay of most of the universe's baryonic matter and then face an eternity of darkness with nothing to touch, it's yet another reason I don't think immortality (as opposed to just a very long lifespan) is actually desirable.
- imtringued 3y ago
  Longtermists argue that we will be harvesting hawking radiation from blackholes trillions of years after the heat death of the universe.
  - __loam 3y ago
    The last civilizations will be built around black holes.
- mewpmewp2 3y ago
  I imagine there would be perhaps tech or technique which you can choose to determine which memories to compress and countless of others techniques like extra storage that you can instantly access, so I don't see all of these as being real arguments why not become immortal. If I have to choose to be dead and memoryless compared to losing some of my memories, but being still alive, why should I choose being dead and memoryless? And when losing memories you would first just discard some details, like you lose now anyway, but you would start compressing centuries into rough ideas of what happened, it's just the details that would lack a bit. I don't see it being a problem at all. And if really something happens with the Universe, sure I can die then, but why would I want to die before? I want to know what happens, what gets discovered, what happens with humanity, how far do we reach in terms of understanding of what is going on in this place. Why are we here. Imagine dying and not even knowing why you were here.
halflings 3y ago
> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everything perfectly, we would not learn anything: instead of remembering the concept of a "chair", you would remember thousands of separate instances of things you've seen that have a certain combination of colors and shapes etc It's the fact that we forget certain details (small differences between all these chairs) that makes us learn what a "chair" is. Likewise, if you remembered every single word in a book, you would not understand its meaning; understanding its meaning = being able to "summarize" (compress) this long list of words into something more essential: storyline, characters, feelings, etc.
- JieJie 3y ago
  My mind is a blurry jpeg of my life. (https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...)
- WanderPanda 3y ago
  Compression = Intelligence http://prize.hutter1.net/ http://prize.hutter1.net/
  - briansm 3y ago
    Nope, more specifically: Lossy compression = Intelligence That's where the Hutter Prize falls down, it's based on lossless compression, which is nothing like how the brain works.
- ComputerGuru 3y ago
  > but that's exactly what OP said, no? Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”. > Likewise, if you remembered every single word in a book, you would not understand its meaning I understand your meaning but I want to clarify for the sake of the discussion that unlike with ML, the human brain can both memorize verbatim and understand the meaning because there is no mechanism for memorizing something but not processing it (i.e. purely storage). The first pass(es) are stripped to their essentials but subsequent passes provide the ability to memorize the same input.
gattilorenz 3y ago
Is there a “realistic upper bound” in things that should be memorized verbatim? Ancient greeks probably memorized the Iliad and other poems (rhyming and metre might work as a substitute for data compression, in this case), and many medieval preachers apparently memorized the whole Bible…
- deleted 3y ago
  [deleted]
TheRealSteel 3y ago
You seem to have just re-stated what the other person said.
- whimsicalism 3y ago
  Thank you, thought I was losing it for a second
firecall 3y ago
Does the brain require more energy to store more information? Or is it always running at the same pace regardless of if it’s empty or not? I guess the Brian doesn’t really work like that…. But I’m curious :-)
- ReactiveJelly 3y ago
  The brain doesn't seem to ever "clock down" when it's idle, which is interesting to me https://en.wikipedia.org/wiki/Human_brain#Metabolism https://en.wikipedia.org/wiki/Human_brain#Metabolism > The energy consumption of the brain does not vary greatly over time
  - gmadsen 3y ago
    maybe between sleep and normal waking idle because there is actually quite a bit going on during sleep. There has been quite a bit of research though regarding higher "clock up" states consuming far more energy, such as grandmasters playing a chess tournament
    - lobocinza 3y ago
      My lazy research pointed to 6K calories which is a lot more than a typical person can burn by physical exercise.
oneTbrain23 3y ago
You obviously hand wave alzheimer and dementia. Human don't know exactly how brains works. The computational storage is just an estimate of what we understand von Neuman computer storing data 1 and 0. In every psychological test conducted on human mind, they clearly have a limit.

downboots 3y ago

Can "distill knowledge" be made precise ?

ComputerGuru 3y ago
As best as I’ve been able to research, it’s still under active exploration and there are hypotheses but no real answers. I believe research has basically been circling around the recent understanding that in addition to being part of how the brain is wired, it is also an active, deliberate (if unconscious) mechanism that takes place in the background and is run “at a higher priority” during sleep (sort of like an indexing daemon running at low priority during waking hours then getting the bulk of system resources devoted to it during idle). There are also studies that show “data” in the brain isn’t stored read-only and the process of accessing that memory involves remapping the neurons (which is how fake memories are possible) - so my take is if you access a memory or datum sequentially start to finish each time the brain knows this is to be stored verbatim for as-is retrieval but if you access snapshots of it or actively seek to and replay a certain part while trying to relate that memory to a process or a new task, the brain rewires the neural pathways accusingly. Which implies that there us an unconscious part that takes place globally plus an active, modifying process where how we use a stored memory affects how it is stored and indexed (so data isn’t accessed by simple fields but rather by complex properties or getters, in programming parlance). I guess the key difference from how machine learning works (and I believe an integral part of AGI, if it is even possible) is that inference is constant, even when you’re only “looking up” data and you don’t know the right answer (i.e. not training stage). The brain recognizes how the new query differs from queries it has been trained on and can modify its own records to take into account the new data. For example, let’s say you’re trying to classify animals into groups and you’ve “been trained” on a dataset that doesn’t include monotremes or marsupials. The first time you come across a platypus in the wild (with its mammaries but no nipples, warm-blooded but lays eggs, and a single duct for waste and reproduction) you wouldn’t just mistakenly classify it as a bird or mammal - you would actively trigger a (delayed/background) reclassification of all your existing inferences to account for this new phenomenon, even though you don’t know what the answer to the platypus classification question is.
clord 3y ago
imo, it amounts to revisiting concepts once more general principles are found — and needed. For instance, you learn the alphabet, and it's hard. the order is tricky. the sounds are tricky, etc. but eventually, it get distilled to a pattern. But you still have to start from A to remember what letter 6 is, until you encounter that problem many times, and then the brain creates a 6=F mapping. I think of it in economic terms: when the brain realizes it's cheaper to create a generalization, it does so on the fly, and that generalization takes over the task. Somtimes it's almost like creating a specialist shard to take over the task. Driving is hard at first, with very high task overload, lots to pay attention to. With practice, it becomes a little automated part of yourself takes care of those tasks while your main general intelligence can do whatever it likes, even as the "driver" deals with seriously difficult tasks.
- pyinstallwoes 3y ago
  It’s almost like a tuning fork. When the mapping becomes 6=F the two separate tuning forms are now sympathetic to a frequency.
esafak 3y ago
https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory
__loam 3y ago
Unless you know something the neuroscientists don't, it cannot.

jjk166 3y ago

Distilling knowledge is data compression.

w10-1 3y ago
You're conflating memorization with generalization, no?
- jjk166 3y ago
  Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.
  - staunton 3y ago
    > Generalization is developing the heuristics by which you compress stored data such that > you nevertheless can recover enough information to be useful in the future. I disagree (in case you meant to imply it) that compression implies generalization.
    - jjk166 3y ago
      I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.

nonameiguess 3y ago

I've thought about this a lot in the context of the desire people seem to have to try and achieve human immortality or at least indefinite lifespans. If SciAm is correct here and the upper bound is a quadrillion bytes, we may not be able to hit that given the bound on possible human experiences, but someone who lived long enough would eventually hit that. After a hundred million years or whatever the real number is of life, you'd either lose the ability to form new memories or you'd have to overwrite old ones to do so. Aside from having to eventually experience the death of all stars and light and the decay of most of the universe's baryonic matter and then face an eternity of darkness with nothing to touch, it's yet another reason I don't think immortality (as opposed to just a very long lifespan) is actually desirable.

imtringued 3y ago
Longtermists argue that we will be harvesting hawking radiation from blackholes trillions of years after the heat death of the universe.
- __loam 3y ago
  The last civilizations will be built around black holes.
mewpmewp2 3y ago
I imagine there would be perhaps tech or technique which you can choose to determine which memories to compress and countless of others techniques like extra storage that you can instantly access, so I don't see all of these as being real arguments why not become immortal. If I have to choose to be dead and memoryless compared to losing some of my memories, but being still alive, why should I choose being dead and memoryless? And when losing memories you would first just discard some details, like you lose now anyway, but you would start compressing centuries into rough ideas of what happened, it's just the details that would lack a bit. I don't see it being a problem at all. And if really something happens with the Universe, sure I can die then, but why would I want to die before? I want to know what happens, what gets discovered, what happens with humanity, how far do we reach in terms of understanding of what is going on in this place. Why are we here. Imagine dying and not even knowing why you were here.

halflings 3y ago

> That’s not exactly true [...] Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components ...but that's exactly what OP said, no? I remember attending an ML presentation where the speaker shared a quote I can't find anymore (speaking of memory and generalization :)), which said something like: "To learn is to forget" If we memorized everything perfectly, we would not learn anything: instead of remembering the concept of a "chair", you would remember thousands of separate instances of things you've seen that have a certain combination of colors and shapes etc It's the fact that we forget certain details (small differences between all these chairs) that makes us learn what a "chair" is. Likewise, if you remembered every single word in a book, you would not understand its meaning; understanding its meaning = being able to "summarize" (compress) this long list of words into something more essential: storyline, characters, feelings, etc.

JieJie 3y ago
My mind is a blurry jpeg of my life. (https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...)
WanderPanda 3y ago
Compression = Intelligence http://prize.hutter1.net/ http://prize.hutter1.net/
- briansm 3y ago
  Nope, more specifically: Lossy compression = Intelligence That's where the Hutter Prize falls down, it's based on lossless compression, which is nothing like how the brain works.
ComputerGuru 3y ago
> but that's exactly what OP said, no? Not precisely. We don’t know if verbatim capacity is limited (and it doesn’t seem to be) but the brain operates in a space-efficient manner all the same. So there isn’t necessarily a causative relationship between “memory capacity” and “means of storage”. > Likewise, if you remembered every single word in a book, you would not understand its meaning I understand your meaning but I want to clarify for the sake of the discussion that unlike with ML, the human brain can both memorize verbatim and understand the meaning because there is no mechanism for memorizing something but not processing it (i.e. purely storage). The first pass(es) are stripped to their essentials but subsequent passes provide the ability to memorize the same input.

gattilorenz 3y ago

Is there a “realistic upper bound” in things that should be memorized verbatim? Ancient greeks probably memorized the Iliad and other poems (rhyming and metre might work as a substitute for data compression, in this case), and many medieval preachers apparently memorized the whole Bible…

deleted 3y ago
[deleted]

TheRealSteel 3y ago

You seem to have just re-stated what the other person said.

whimsicalism 3y ago
Thank you, thought I was losing it for a second

firecall 3y ago

Does the brain require more energy to store more information? Or is it always running at the same pace regardless of if it’s empty or not? I guess the Brian doesn’t really work like that…. But I’m curious :-)

ReactiveJelly 3y ago
The brain doesn't seem to ever "clock down" when it's idle, which is interesting to me https://en.wikipedia.org/wiki/Human_brain#Metabolism https://en.wikipedia.org/wiki/Human_brain#Metabolism > The energy consumption of the brain does not vary greatly over time
- gmadsen 3y ago
  maybe between sleep and normal waking idle because there is actually quite a bit going on during sleep. There has been quite a bit of research though regarding higher "clock up" states consuming far more energy, such as grandmasters playing a chess tournament
  - lobocinza 3y ago
    My lazy research pointed to 6K calories which is a lot more than a typical person can burn by physical exercise.

oneTbrain23 3y ago

You obviously hand wave alzheimer and dementia. Human don't know exactly how brains works. The computational storage is just an estimate of what we understand von Neuman computer storing data 1 and 0. In every psychological test conducted on human mind, they clearly have a limit.