Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is N

Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is NLP or CV. For instance, in physical simulations, the representation of the system you’re simulating could be on the order of 10s of gigabytes (a very large mesh, a fine 3D grid, etc.) These are the kinds of problems I work on, and we massively benefit every time a new chip comes out with more memory or increased memory bandwidth, because a single training example, plus it’s activations, the model, and optimizer state, can take about a half a TB of memory or more depending on model size. I’d also point out that as hardware devices get better, it’s good for everyone. It allows the large corporations to train even larger, more capable models, and it allows smaller players to train models they couldn’t have afforded before on smaller budgets. A rising tide lifts all boats, so to speak. Algorithmic and efficiency improvements are also important, but they’re additive, not a replacement. And while combining models is not novel and is certainly an avenue that should and is being explored, the models that have the most generalizability for recombination are these extremely large models! They have the capacity to learn very generalized patterns that are then useful for far more transfer/combination tasks.