The fact of the matter is that bigger networks just often do a lot better. Much bigger networks, much more so. So I think, until some group of geniuses comes up

The fact of the matter is that bigger networks just often do a lot better. Much bigger networks, much more so. So I think, until some group of geniuses comes up with some radical optimization algorithm that rapidly trains massive sets of matrices, we're in an era of hardware-/bandwidth-/electricity-limited improvements. I kinda think that's why LightOn is so interesting; they're trying to do neural networks by casting the matrix multiplies as doing optical stuff with photons (if I understand correctly, some kind of compressed sensing training method involving scattering randomized apertures for light). I don't know much about their tech, but that seems like a paradigm shift compared to GPU cores.. they purport 3- Peta multiply-accumulate ops per second, at 30 watts. https://lighton.ai/our-technology/#1596017080086-e6716927-baea https://lighton.ai/our-technology/#1596017080086-e6716927-ba... Would like to see the benchmarks though, their benchmarks repo doesn't just have it immediately all laid out