3 ms·
~85% accuracy on MNIST. Sigh. How does it do on CIFAR-10, or even better, ImageNet? Interesting research, not sure it's a backprop alternative. === EDIT: ac
by cs702 19d ago
~85% accuracy on MNIST. Sigh.
How does it do on CIFAR-10, or even better, ImageNet?
Interesting research, not sure it's a backprop alternative.
===
EDIT: accuracy on MNIST is not ~90%. It's ~85%.
- bz_bz_bz 19d agoTheir image classification benchmarks include both: https://pub.sakana.ai/pc-alm/assets/figures/benchmark_accuracy_depths.webp https://pub.sakana.ai/pc-alm/assets/figures/benchmark_accura...
- cs702 19d ago~74% on CIFAR-10. Still a far cry from backprop. I didn't see ImageNet. TinyImageNet is something else.
- rocketships 19d agoThere might be only one paper that's trained on full Imagenet using methods like these. Training a Predictive Coding Network on ImageNet using Equilibrium Propagation Tugdual Kerjan, Rasmus Høier, Benjamin Scellier https://arxiv.org/abs/2606.03584 https://arxiv.org/abs/2606.03584 It's quite an undertaking.
- Lerc 19d agoIt might be beneficial while not being optimal on its own. The obvious example is if it has different behaviour around local minima, it could be an altenate pathway out. I have often wondered if doing training with radically different aproaches for the first few iterarions would avoid any method specific artifacts before the weights had time to denoise.
- qarl 19d agoThey state replacing backprop is not their goal. Their goal is to understand how distributed systems which cannot do backprop (the brain) can still do learning.
- im3w1l 19d agoPersonally I think the dirty secret of the brain is that a lot of things are hard coded. And many things that we need to learn are also hard coded except that some parameters need to be tuned. If we puke, the brain will not do general aversive learning, it will learn to avoid specifically the last thing eaten, because it instinctively knows about food poisoning. Imprinting is absolutely fascinating. Some newborn animals will run a very simple pattern detector like looking for a red dot or something and use that to bootstrap their conception of their parent. For fully general learning I have a hunch that it can be done using local history plus a semi-global reward scalar (global neurotransmittor levels).
- DoctorOetker 19d agoregardless if the intelligence in the brain is hardcoded or not, to the extent it is, this information must have been compressed in the genome, which runs counter to almost all observations: a child doesn't remember the experience of their ancestors, for example. The only sense in which we do carry mental state without relearning is emotions, instincts, reflexes (some neuronal pathways that connect the eye to the middle ear), hormonal driven behavior (fear adrenalin). For another, there are about 200k promotor regions (including non-coding) in the human genome. A promotor region might have say 6 to 15 bits of information. Can you compress 2025 or even 2024 era LLM intelligence into 3 megabit = ~400 kB ? I think not. I think a lot of compression is still possible, but 400 kB? So I think we can box up the idea of "dirty secrets of the braing: not learning but hard coding". There is a lot of hard coding in biology, but brains are evolved specifically to enable learning within the individual lifetime instead of only learning by natural selection. I also don't buy the following argument: > If we puke, the brain will not do general aversive learning, it will learn to avoid specifically the last thing eaten, because it instinctively knows about food poisoning. Each time it happens that I end up puking, I do feel aversion and try to avoid puking at all, sometimes I succeed but sometimes is just puke. There must be fundamental puke reflexes (which one fails to avoid) and avertable puke reflexes.
- imtringued 19d agoIt's really sad that they are only a few years away from making backpropagation completely obsolete.