3 ms·
And there is no particular reason to believe that capabilities are merely the sum of a bunch of random iid variables! If you look at how DL scales with model si
by gwern 6y ago
And there is no particular reason to believe that capabilities are merely the sum of a bunch of random iid variables! If you look at how DL scales with model size or data or FLOPS, it's a powerlaw (or lognormal), and in countless places in the natural world, things are lognormal rather than normal. Similarly, wealth, scientific productivity, lots of things are lognormal rather than normal. Even for human intelligence, the normality is due more to the fact that we all come off the assembly line as Mark 1 human units, practically identical, with the differences coming from environmental randomness & randomness in genetic mutation assortment.