3 ms·

> If someone's hippocampus is destroyed, they lose the ability to form new (episodic) memories, and may lose some more recent old ones, but they certainly do no

> If someone's hippocampus is destroyed, they lose the ability to form new (episodic) memories, and may lose some more recent old ones, but they certainly do not lose older ones. This is basic knowledge. Yes, like basic reading without digging into details. You must have heard about HM, since you’re saying all this. But here’s the facts: When H.M.’s remote memories were probed carefully, they turned out to be gist-like and semanticized, not vivid re-experiencings of specific events. Here’s the paper: https://pubmed.ncbi.nlm.nih.gov/15716139/ https://pubmed.ncbi.nlm.nih.gov/15716139/ Only semantic memory of the episodes can be said to be “cortical” (though please note, lack of hippocampus doesn’t mean lack of other brain regions…). Rich recall absolutely does require the hippocampus. Once again, please try not to “spherical cow” the complexity of the brain to try and fit it into your analogy to digital computing. You will get an underdermined model that will miss the subtleties, and lead you to claims that are poor fits for the reality. As for why I brought up bird brains… there part of the same evolutionary web. Is there some reason you want them excluded? They’re a well studied model for a fairly complex memory task, using substantially smaller neurons more densely packed in a different architecture than mammals. In cognitive science, as in computer science I’d imagine, it’s useful to look at the full picture before making strong claims. The bird case is interesting because the region of interest is a nucleus, rather than cortical columns, and actually has well documented structural variations in size, as well as gene expression, over the seasons, while the memories are forming. If your model is correct, it needs to account for those facts.

HarHarVeryFunny 23d ago
> If your model is correct, it needs to account for those facts. Let me make it simple for you. We have a finite number of neurons in our brain, as do birds, and our brain is attempting to store an ever growing number of memories in those. And, no, this is not a digital computer (is your reading comprehension really so bad?). Nobody, including you, knows exactly where different types of memory are stored, and for my argument it makes no difference. What does make a difference is how they are represented, which I am suggesting is sparse embeddings. So far, you've been ignoring my actual argument and instead responding to various strawmen of your own making, so it's not clear if you even understand what a sparse embedding is. If you do understand, then it should be obvious that it makes no difference whether the neurons comprising this embedding space are in the hippocampus, cortex/pallium or anywhere else. Clearly you do NOT understand, since you bring up bird brains (pallium vs cortex) and want to argue about location of storage (hippocampus vs elsewhere) as if it made a difference to MY argument. My argument (if you care to respond to it, which so far you have not) is that when sparse embeddings have little to no overlap, then associative recall by a similar pattern will work reliably, but when multiple embeddings have too much in common then recall will suffer as multiple embeddings will match. Hint: if you think this has anything to do with digital computers then you have misunderstood and need to go back and re-read more carefully, or google for any terms you do not understand.
- randomImmigrant 23d ago
  Sliding past the mistakes pointed out, shifting goalposts and trying to recover I see. Let’s say I’m a complete moron and don’t know what a sparse embedding is. Pretty please, can you define it for me and then tell me, in detail, where in whatever region of the brain you think this is going on… how is it going on? Explain how “memories must be stored as embeddings with single multi-neuron assemblies (cortical columns?) storing multiple embeddings as a kind of contents-addressable memory” You have moved past the cortical column. But still seem to be insisting it’s a bunch of neurons, somewhere… or has that also conveniently changed? Whatever your current position is, please go ahead and explain what components of what cells or otherwise are involved in this process you’re describing.
  - HarHarVeryFunny 22d ago
    An embedding space is a (typically) high dimensional space that has enough dimensions such that examples of some type of entity (e.g. faces, words, or thoughts) can be represented as points in that space, positioned such that they are nearby to other entities with which they have things in common. An entity embedding doesn't need to use all the dimensions of the space it is positioned in - some dimensions may be unused (sometimes represented as a coorrdinate of 0 in that dimension). These are called "sparse" embeddings. For example, an LLM's tokens are represented as embeddings in what is typcially an approximately ~1000 dimensional space, but start out as sparse embeddings just representing a short letter sequence (but then go on to be transformed/augmented with additional information and so become less sparse). As an example, let's say an embedding space has 10 dimensions, then a couple of sparse embedding examples could be: [0 0 1 0 0 1 1 0 0 0] [1 1 0 0 0 0 0 1 0 0] These two embeddings have no overlap (where both are non-zero), and the more dimensions you have the more likely it is that two random sparse embedding will have little in common. Embeddings are used in many types of artificial neural networks, not just LLMs, for example face recognition networks, where they are trained such that similar faces (multiple photos of the same person) are close together in the embedding space, and post-training you can then "look up" any arbitrary photo (in the training set or not) by embedding it and seeing what is nearby in the embedding space, which will be similar looking faces. Presumably real neural networks used embeddings in a similar way, since, for example, it obviously requires many neurons to represent the many differences between different faces, and there is going to be overlap between the neurons used to represent multiple faces (this is not a computer with one storage location for face #1, and a different location for face #2). A neural network, real or artificial, uses groups of neurons (e.g. a cortical column) to represent an embedding space, with each neuron corresponding to a dimension. A single group of neurons (column) can store multiple embeddings (e.g. faces) represented as different activity patterns (which neurons are firing), and if these are sparse embeddings then the firing patterns corresponding to different memories stored in the same column will have little in common. Now, I don't know how you believe associative recall is implemented in the brain - how does someone's voice, or half obscured face, recall their entire face, so feel free to imagine it as implemented however you will, but I'd suggest that in an assembly such as a cortical column that when a set of synaptic inputs are triggered the assembly as a whole will learn to reactivate the entire pattern when only part of the original set of synaptic inputs are triggered, and this is the basis of associative recall. There are papers that suggest exactly how this may work given the cortical column microcircuit. So, with all that said, the suggestion I was making for why (or at least one reason why) memory degrades with age, with memories blending together, is that with a finite quantity of "storage" (cortical columns) you will eventually be storing so many memories (absent a deliberate forgetting mechanism) that there will inevitably be overlap between the sparse embedddings, and this associative recall will therefore not cleanly recall individual memories but rather recall blended memories according to what they have in common. Obviously some types of memory are at least initially stored in the hippocampus, so no reason to focus on cortical columns, but I expect the use of embeddings is universal.
exebook 21d ago
There is 2006 paper "Polychronization: Computation with Spikes" E. Izhikevich. that describes one simulation they have done in silico and it explains how exactly distribution of activity happens and why you can't run out of memory. Basically the small group of neurons, say 10, can represent much larger amount of information say 1000 because they can fire in different orders, that is what they call poly-synchronous activity.

HarHarVeryFunny 23d ago

> If your model is correct, it needs to account for those facts. Let me make it simple for you. We have a finite number of neurons in our brain, as do birds, and our brain is attempting to store an ever growing number of memories in those. And, no, this is not a digital computer (is your reading comprehension really so bad?). Nobody, including you, knows exactly where different types of memory are stored, and for my argument it makes no difference. What does make a difference is how they are represented, which I am suggesting is sparse embeddings. So far, you've been ignoring my actual argument and instead responding to various strawmen of your own making, so it's not clear if you even understand what a sparse embedding is. If you do understand, then it should be obvious that it makes no difference whether the neurons comprising this embedding space are in the hippocampus, cortex/pallium or anywhere else. Clearly you do NOT understand, since you bring up bird brains (pallium vs cortex) and want to argue about location of storage (hippocampus vs elsewhere) as if it made a difference to MY argument. My argument (if you care to respond to it, which so far you have not) is that when sparse embeddings have little to no overlap, then associative recall by a similar pattern will work reliably, but when multiple embeddings have too much in common then recall will suffer as multiple embeddings will match. Hint: if you think this has anything to do with digital computers then you have misunderstood and need to go back and re-read more carefully, or google for any terms you do not understand.

randomImmigrant 23d ago
Sliding past the mistakes pointed out, shifting goalposts and trying to recover I see. Let’s say I’m a complete moron and don’t know what a sparse embedding is. Pretty please, can you define it for me and then tell me, in detail, where in whatever region of the brain you think this is going on… how is it going on? Explain how “memories must be stored as embeddings with single multi-neuron assemblies (cortical columns?) storing multiple embeddings as a kind of contents-addressable memory” You have moved past the cortical column. But still seem to be insisting it’s a bunch of neurons, somewhere… or has that also conveniently changed? Whatever your current position is, please go ahead and explain what components of what cells or otherwise are involved in this process you’re describing.
- HarHarVeryFunny 22d ago
  An embedding space is a (typically) high dimensional space that has enough dimensions such that examples of some type of entity (e.g. faces, words, or thoughts) can be represented as points in that space, positioned such that they are nearby to other entities with which they have things in common. An entity embedding doesn't need to use all the dimensions of the space it is positioned in - some dimensions may be unused (sometimes represented as a coorrdinate of 0 in that dimension). These are called "sparse" embeddings. For example, an LLM's tokens are represented as embeddings in what is typcially an approximately ~1000 dimensional space, but start out as sparse embeddings just representing a short letter sequence (but then go on to be transformed/augmented with additional information and so become less sparse). As an example, let's say an embedding space has 10 dimensions, then a couple of sparse embedding examples could be: [0 0 1 0 0 1 1 0 0 0] [1 1 0 0 0 0 0 1 0 0] These two embeddings have no overlap (where both are non-zero), and the more dimensions you have the more likely it is that two random sparse embedding will have little in common. Embeddings are used in many types of artificial neural networks, not just LLMs, for example face recognition networks, where they are trained such that similar faces (multiple photos of the same person) are close together in the embedding space, and post-training you can then "look up" any arbitrary photo (in the training set or not) by embedding it and seeing what is nearby in the embedding space, which will be similar looking faces. Presumably real neural networks used embeddings in a similar way, since, for example, it obviously requires many neurons to represent the many differences between different faces, and there is going to be overlap between the neurons used to represent multiple faces (this is not a computer with one storage location for face #1, and a different location for face #2). A neural network, real or artificial, uses groups of neurons (e.g. a cortical column) to represent an embedding space, with each neuron corresponding to a dimension. A single group of neurons (column) can store multiple embeddings (e.g. faces) represented as different activity patterns (which neurons are firing), and if these are sparse embeddings then the firing patterns corresponding to different memories stored in the same column will have little in common. Now, I don't know how you believe associative recall is implemented in the brain - how does someone's voice, or half obscured face, recall their entire face, so feel free to imagine it as implemented however you will, but I'd suggest that in an assembly such as a cortical column that when a set of synaptic inputs are triggered the assembly as a whole will learn to reactivate the entire pattern when only part of the original set of synaptic inputs are triggered, and this is the basis of associative recall. There are papers that suggest exactly how this may work given the cortical column microcircuit. So, with all that said, the suggestion I was making for why (or at least one reason why) memory degrades with age, with memories blending together, is that with a finite quantity of "storage" (cortical columns) you will eventually be storing so many memories (absent a deliberate forgetting mechanism) that there will inevitably be overlap between the sparse embedddings, and this associative recall will therefore not cleanly recall individual memories but rather recall blended memories according to what they have in common. Obviously some types of memory are at least initially stored in the hippocampus, so no reason to focus on cortical columns, but I expect the use of embeddings is universal.

exebook 21d ago

There is 2006 paper "Polychronization: Computation with Spikes" E. Izhikevich. that describes one simulation they have done in silico and it explains how exactly distribution of activity happens and why you can't run out of memory. Basically the small group of neurons, say 10, can represent much larger amount of information say 1000 because they can fire in different orders, that is what they call poly-synchronous activity.