7 ms·

Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to c

Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done.

pillefitz 3y ago
That is essentially what embeddings do
- nightski 3y ago
  Maybe, except from my understanding an embedding vector tends to be much larger than the source token (due to the high dimensionality of the embedding space). So it's almost like a reverse compression in a way. That said I know vector DBs have much more efficient ways of storing those vector embedding.
  - jncfhnb 3y ago
    Tokens are not 1:1 with vectors.
ComputerGuru 3y ago
That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-boosts-the-human-brain-s-memory-capacity-10-fold/ https://www.scientificamerican.com/article/new-estimate-boos...
- downboots 3y ago
  Can "distill knowledge" be made precise ?
  - ComputerGuru 3y ago
    As best as I’ve been able to research, it’s still under active exploration and there are hypotheses but no real answers. I believe research has basically been circling around the recent understanding that in addition to being part of how the brain is wired, it is also an active, deliberate (if unconscious) mechanism that takes place in the background and is run “at a higher priority” during sleep (sort of like an indexing daemon running at low priority during waking hours then getting the bulk of system resources devoted to it during idle). There are also studies that show “data” in the brain isn’t stored read-only and the process of accessing that memory involves remapping the neurons (which is how fake memories are possible) - so my take is if you access a memory or datum sequentially start to finish each time the brain knows this is to be stored verbatim for as-is retrieval but if you access snapshots of it or actively seek to and replay a certain part while trying to relate that memory to a process or a new task, the brain rewires the neural pathways accusingly. Which implies that there us an unconscious part that takes place globally plus an active, modifying process where how we use a stored memory affects how it is stored and indexed (so data isn’t accessed by simple fields but rather by complex properties or getters, in programming parlance). I guess the key difference from how machine learning works (and I believe an integral part of AGI, if it is even possible) is that inference is constant, even when you’re only “looking up” data and you don’t know the right answer (i.e. not training stage). The brain recognizes how the new query differs from queries it has been trained on and can modify its own records to take into account the new data. For example, let’s say you’re trying to classify animals into groups and you’ve “been trained” on a dataset that doesn’t include monotremes or marsupials. The first time you come across a platypus in the wild (with its mammaries but no nipples, warm-blooded but lays eggs, and a single duct for waste and reproduction) you wouldn’t just mistakenly classify it as a bird or mammal - you would actively trigger a (delayed/background) reclassification of all your existing inferences to account for this new phenomenon, even though you don’t know what the answer to the platypus classification question is.
  - 3y ago
bufferoverflow 3y ago
There are rare people who remember everything https://youtu.be/hpTCZ-hO6iI https://youtu.be/hpTCZ-hO6iI
- svachalek 3y ago
  It's pretty fascinating to me how "normal" Marilu Henner seems to be. I'm getting older and my memory is not what it was, but when I was younger it was pretty extraordinary. I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly, over being truly "smart" in a classic sense. But having so much of the past being so accessible is tough. There are lots of memories I'd rather not have, that are vivid and easily called up. And still, I think it's only a fraction of what her memory seems to be like.
  - 93po 3y ago
    As someone on the other end of the spectrum, I have an awful memory, and don't remember most of my life aside from really wide, sweeping generalizations and maybe a couple hundred very specific memories. My way of existence is also very sad, and it makes me feel like I've not really lived.
    - obscurette 3y ago
      It's likely that you actually have memories about details, but don't have a way to recall these memories. I always wondered how the heck people write memories until I saw someone to do it. He used a lot of triggers – photos, newspapers, letters etc. Later I had a chance to visit museum where typical home environment of my childhood was exhibited (yes, I'm that old) and realized how many memories small things can trigger in my brain.
      - 93po 3y ago
        I agree. There are definitely triggers for old memories and I have heard cannabis also adds flexibility to that recall
bobboies 3y ago
Good example in my math and physics classes I found it really helpful to understand the general concepts, then instead of memorizing formulas could actually derive them from other known (perhaps easier-to-remember) facts. Geometry is good for training in this way—and often very helpful for physics proofs too!
- lacrimacida 3y ago
  Too bad this method is penalized most on tests (timed) where memorization is favored. But deriving results reinforce knowledge, understanding and patterns best in my opinion.
BSEdlMMldESB 3y ago
yes, when we do this to history, it becomes filled with conspiracies. but is merely a process to 'understand' history by projecting intentionalities. this 'compression' is what 'understanding' something really entails; at first... but then there's more. when knowledge becomes understood it enables perception (e.g. we perceive meaning in words once we learn to read). when we get really good at this understanding-perception we may start to 'manipulate' the abstractions we 'perceive'. an example would be to 'understand a cube' and then being able to rotate it around so to predict what would happen without really needing the cube. but this is an overly simplistic example
- NovaDudely 3y ago
  This was the thinking I was taking. It is a useful tool at first but taken too far can be a bad thing in some situations.
tbalsam 3y ago
For more information and the related math behind associative memories, please see Hopfield Neural Networks. While the upper bound is technically "infinity", there is a tradeoff between the amount of concepts stored and the fundamental amount of information storable per concept, similar to how other tradeoff principles like the uncertainty principle, etc work.
- scrps 3y ago
  Thank you
pyinstallwoes 3y ago
Maxwell’s demon to entropy
mr_toad 3y ago
Artificial neural networks work a lot like compression algorithms in their ability to predict the future. The trained network is a compression algorithm - it does not store compressed data. We don’t know if the animal brain works the same way, but I suspect it is mostly compression algorithms designed to predict things, and doesn’t store much data at all.

pillefitz 3y ago

That is essentially what embeddings do

nightski 3y ago
Maybe, except from my understanding an embedding vector tends to be much larger than the source token (due to the high dimensionality of the embedding space). So it's almost like a reverse compression in a way. That said I know vector DBs have much more efficient ways of storing those vector embedding.
- jncfhnb 3y ago
  Tokens are not 1:1 with vectors.

ComputerGuru 3y ago

That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-boosts-the-human-brain-s-memory-capacity-10-fold/ https://www.scientificamerican.com/article/new-estimate-boos...

downboots 3y ago
Can "distill knowledge" be made precise ?
- ComputerGuru 3y ago
  As best as I’ve been able to research, it’s still under active exploration and there are hypotheses but no real answers. I believe research has basically been circling around the recent understanding that in addition to being part of how the brain is wired, it is also an active, deliberate (if unconscious) mechanism that takes place in the background and is run “at a higher priority” during sleep (sort of like an indexing daemon running at low priority during waking hours then getting the bulk of system resources devoted to it during idle). There are also studies that show “data” in the brain isn’t stored read-only and the process of accessing that memory involves remapping the neurons (which is how fake memories are possible) - so my take is if you access a memory or datum sequentially start to finish each time the brain knows this is to be stored verbatim for as-is retrieval but if you access snapshots of it or actively seek to and replay a certain part while trying to relate that memory to a process or a new task, the brain rewires the neural pathways accusingly. Which implies that there us an unconscious part that takes place globally plus an active, modifying process where how we use a stored memory affects how it is stored and indexed (so data isn’t accessed by simple fields but rather by complex properties or getters, in programming parlance). I guess the key difference from how machine learning works (and I believe an integral part of AGI, if it is even possible) is that inference is constant, even when you’re only “looking up” data and you don’t know the right answer (i.e. not training stage). The brain recognizes how the new query differs from queries it has been trained on and can modify its own records to take into account the new data. For example, let’s say you’re trying to classify animals into groups and you’ve “been trained” on a dataset that doesn’t include monotremes or marsupials. The first time you come across a platypus in the wild (with its mammaries but no nipples, warm-blooded but lays eggs, and a single duct for waste and reproduction) you wouldn’t just mistakenly classify it as a bird or mammal - you would actively trigger a (delayed/background) reclassification of all your existing inferences to account for this new phenomenon, even though you don’t know what the answer to the platypus classification question is.
- 3y ago

bufferoverflow 3y ago

There are rare people who remember everything https://youtu.be/hpTCZ-hO6iI https://youtu.be/hpTCZ-hO6iI

svachalek 3y ago
It's pretty fascinating to me how "normal" Marilu Henner seems to be. I'm getting older and my memory is not what it was, but when I was younger it was pretty extraordinary. I did really well in school and college but over time I've realized it was mostly due to being able to remember most things pretty effortlessly, over being truly "smart" in a classic sense. But having so much of the past being so accessible is tough. There are lots of memories I'd rather not have, that are vivid and easily called up. And still, I think it's only a fraction of what her memory seems to be like.
- 93po 3y ago
  As someone on the other end of the spectrum, I have an awful memory, and don't remember most of my life aside from really wide, sweeping generalizations and maybe a couple hundred very specific memories. My way of existence is also very sad, and it makes me feel like I've not really lived.
  - obscurette 3y ago
    It's likely that you actually have memories about details, but don't have a way to recall these memories. I always wondered how the heck people write memories until I saw someone to do it. He used a lot of triggers – photos, newspapers, letters etc. Later I had a chance to visit museum where typical home environment of my childhood was exhibited (yes, I'm that old) and realized how many memories small things can trigger in my brain.
    - 93po 3y ago
      I agree. There are definitely triggers for old memories and I have heard cannabis also adds flexibility to that recall

bobboies 3y ago

Good example in my math and physics classes I found it really helpful to understand the general concepts, then instead of memorizing formulas could actually derive them from other known (perhaps easier-to-remember) facts. Geometry is good for training in this way—and often very helpful for physics proofs too!

lacrimacida 3y ago
Too bad this method is penalized most on tests (timed) where memorization is favored. But deriving results reinforce knowledge, understanding and patterns best in my opinion.

BSEdlMMldESB 3y ago

yes, when we do this to history, it becomes filled with conspiracies. but is merely a process to 'understand' history by projecting intentionalities. this 'compression' is what 'understanding' something really entails; at first... but then there's more. when knowledge becomes understood it enables perception (e.g. we perceive meaning in words once we learn to read). when we get really good at this understanding-perception we may start to 'manipulate' the abstractions we 'perceive'. an example would be to 'understand a cube' and then being able to rotate it around so to predict what would happen without really needing the cube. but this is an overly simplistic example

NovaDudely 3y ago
This was the thinking I was taking. It is a useful tool at first but taken too far can be a bad thing in some situations.

tbalsam 3y ago

For more information and the related math behind associative memories, please see Hopfield Neural Networks. While the upper bound is technically "infinity", there is a tradeoff between the amount of concepts stored and the fundamental amount of information storable per concept, similar to how other tradeoff principles like the uncertainty principle, etc work.

scrps 3y ago
Thank you

pyinstallwoes 3y ago

Maxwell’s demon to entropy

mr_toad 3y ago

Artificial neural networks work a lot like compression algorithms in their ability to predict the future. The trained network is a compression algorithm - it does not store compressed data. We don’t know if the animal brain works the same way, but I suspect it is mostly compression algorithms designed to predict things, and doesn’t store much data at all.