It really makes me appreciate how “brute force” like NNs are, especially back when MLPs were the cool kids. Even today, they are seriously compute constrained a

It really makes me appreciate how “brute force” like NNs are, especially back when MLPs were the cool kids. Even today, they are seriously compute constrained and memory constrained. Bigger actually is better. There’s all this debate about how things sold work and ideas going in and out of fashion, but once chips are strong enough, a lot of that doesn’t matter. I’d bet this holds in all sorts of fields. Somebody has a great idea that really needs good steel, and they’re unfashionable (“that’s dumb! It’ll never work (today)!”) until we invent a better steel process. Or ideas that just took a microscope to verify. So many cycles of debate and orthogonal progress rendered unnecessary when a relatively unrelated change happens.