4 ms·
Is there any chance you can answer in simple terms whether the "Hopf coherence" method would be any faster way to do a back propagation than current methods (gr
by rnosov 4y ago
Is there any chance you can answer in simple terms whether the "Hopf coherence" method would be any faster way to do a back propagation than current methods (gradient descent)? My math skills are bit rusty now so I can't tell from looking at your paper alone.
- adamnemecek 4y agoYes, that's the reason why I'm talking about it. It should be faster since it works locally (within the single layers) as opposed to across the whole graph.
- rnosov 4y agoCould you quantify it (Twice, three times, etc)? Another question, does it apply to only attention heads learning or the whole shebang?
- adamnemecek 4y agoIt's going to be more than that, like significantly. Hopf coherence theoretically makes away with backprop altogether. I'm in the process of implementing it but it might take sometime. I'm aware of Hinton's forward-forward, but I need more time to provide a comparison.