4 ms·
Interesting take. Hadn't thought of it in terms of entropy, but it's true. Almost by definition as the training proces doesn't introduce anything novel beyond s
by repelsteeltje 6mo ago
Interesting take. Hadn't thought of it in terms of entropy, but it's true. Almost by definition as the training proces doesn't introduce anything novel beyond scraped inputs and a randomly initialized network. From there, the stochastic generation only adds randomness (and the prompt, of course).
- epr 6mo agoGenerally I think this is a legitemate issue, although: > the training process doesn't introduce anything novel This is not always the case. A compiler, linter, proof checker, tests, etc. can all lower entropy.