The analogy in my mind is the heyday of x86 ubiquity vs the coming revolution of many SoCs in one device. Huge, all-purpose, massive-capacity NNs are the norm

The analogy in my mind is the heyday of x86 ubiquity vs the coming revolution of many SoCs in one device. Huge, all-purpose, massive-capacity NNs are the norm for the current advances in SOTA. This is probably because of a certain limit on development complexity: designing a complex system of richly interacting parts is hard, so as long as development of these sorts of tools is manual, it will be much easier just to shove more compute at the problem with singular, massive models. Compare this to the all-encompassing architecture hegemony of x86. This makes it relatively easy to write software that people will be able to use on devices they already own, and this reduction in developer effort and complexity is enormous for enabling rapid growth in the number of possible software programs that can be created per unit time. As ML design becomes more automated and functional, new possibilities open up regarding breaking out sub-tasks to individual, hyper-specialized tools and combining them into a resilient and capable whole that is more than the sum of its parts. That is the true power of automatic differentiation frameworks: when your system can be end-to-end differentiable throughout many devices and specialized functions, and you can train parts as easily as you can train the whole, you begin to observe the creation -- the growth, really -- of a new kind of digital cognition machine.