As a neurobiologist, this interests me. I'm leaving this comment because I don't have time to read the article rn, but what to come back and see what HN thinks

As a neurobiologist, this interests me. I'm leaving this comment because I don't have time to read the article rn, but what to come back and see what HN thinks about it. Briefly, and naive to the content of the article, here is my concern. Consider this network architecture... You have 3000 input neurons that each makes ~10 synaptic connections with many downstream neurons, but let's focus on just one of those downstream neurons. This downstream neuron thus has 30k synaptic connections along its dendrite. At some time t its cell body (specifically the axon hillock area) receives enough graded electrical potential from some number of those synapses to fire an 'action potential' (a single electrical impulse). Some of those inputs contributed bursts of weak inputs, some may have contributed several strong inputs, some of the inputs were from synapses on very distal regions of the dendrite (therefore the graded signal was significantly weakened by the time it reached the cell body) some were from synapses very near the cell body. But they all integrate at the cell body, at which point, it doesn't matter where they came from - they've arrived is all that matters now - and together their strength is enough to evoke an action potential. Let's say this action potential impulse resulted in an error of some sort, somewhere downstream in the network. If you're playing by the rules of supervised learning with backprop, the synapses that evoked the signal producing the error-impulse should be made weaker. How? In biological NNs this is the impossible question. Signals that could target individual synapses do not propagate up axons through cell bodies up dendrites and back to synapses (you do have slow signaling that is for homeostatic scaling, but this is thought to simply scale network input up or down across all synapses). This means you will need another group of neurons to send error signals back. Maybe these recurrent projections exist, and say they did, these recurrent projections would need to (1) form a 3rd party connection at every synapse and (2) know which synapses were to blame for the error. Those don't seem like trivial phenomena from a neurophysiology perspective. I've read theories on local molecular tags or peptide synthesis, but they all seemed very much hand-wavy. Anything that has gained traction on those hypotheses hasn't stood up to scrutiny. The simplest theory to me, is that cortical networks on the size of human brains don't explicitly 'forget'. That is, they only learn. When you call someone Bob, and they correct you saying their name is Mike, your neural nets don't erase Bob and replace it with Mike. They remember the old name, the new name, the embarrassing situation, all of it. Biological neural nets don't need to explicitly forget because forgetting is (unfortunately?) a byproduct of being a biological entity. Also, neurons have finite resources (they are essentially zero-sum), so you simply will not see infinite run-up of synaptic strengths (which you might in some artificial NN without setting certain hyperparameters). Together, this suggests that recency is a fairly dominant factor when it comes to associative learning (and what gets forgotten) in biological NN.