11 ms·
Because it's all just scaled perceptron isn't it? The reality is that we had to solve vanishing / exploding gradient problem. That wasn't accomplished by getti
by clarionbell 2y ago
Because it's all just scaled perceptron isn't it?
The reality is that we had to solve vanishing / exploding gradient problem. That wasn't accomplished by getting faster compute, but by devising better architectures and using activation functions with properties we needed.
That's how ANNs got out of the "somewhat useful" into "really useful" categories. Hardware helped move it even further. But without the work of computer scientists and mathematicians we would never get to the point when the hardware matters.
- llm_trw 2y agoEven in the bad old days you got your golden tickets every so often. I was working on an alternative approach of using much smoother activation functions at much higher precision, 64 bits in production using GPUs and 128 in testing using CPUs. It worked well enough that I don't think the field would have slowed down much. The only issue is that we'd be working on networks that are 10 times smaller.