3 ms·

That's why the Chinchilla paper (given a single paragraph in the article) is so important; it gives a scaling equation that puts a limit on the effect of increa

That's why the Chinchilla paper (given a single paragraph in the article) is so important; it gives a scaling equation that puts a limit on the effect of increasing parameters. Generally for the known transformer models the reduction in loss from having infinite parameters is significantly less that the reduction in loss from training on infinite data. Most large models are very undertrained.

mjburgess 4y ago
My issue is less with the strategy as NN optimisaiton, and more as the OP says, a road to general intelligence. There's an a priori "limit" on the form of any time/space complexity "intelligence" can have: it cannot be exponential. We know that if you exhaust all infinite number of ways of describing a system by facts (/data), then you don't need intelligence. Whatever intelligence is, it's a strategy animals use for competent action in the face of their bodies not being able to exponentially expend energy to solve every trivial problem. Exponential complexity is a killer to the project of building intelligence in every respect. The universe, a priori, cannot require an exponential space complexity description of it to be parsable: since that would require exponentially many universes. Exponential scaling, for me (via ordinary conceptual analysis), seems a hallmark of any system not using intelligence. Exp. scaling is "the shortcut" to solving the problem without intelligence. It's the hallmark of the system not building O(log_{large number}) rich representations of the world.