2 ms·
The combinatorics on choosing 500 pieces (words) out of a bag of 1.8 million pieces (approx parameters per layer for GPT-3) with replacement, and order matters
by fragmede 2y ago
The combinatorics on choosing 500 pieces (words) out of a bag of 1.8 million pieces (approx parameters per layer for GPT-3) with replacement, and order matters works out to be something like 10^4600. Maybe you can't call that creativity, but you've got to admit that's a pretty big number.
- belter 2y agoI said No handwaving with scale. :-)
- fragmede 2y agoRight—but why should “new ABS plastic” be the bar for creativity? If the kid builds a structure no one’s ever imagined, from an unimaginably large box of Lego, isn’t that still novel? Sure, it’s made from known parts—but so is language. So are most human ideas. The demand for outputs that are provably untraceable to training data feels like asking for magic, not creativity. Even Gödel didn’t require “never seen before atoms” to demonstrate emergence.