3 ms·
Wrt "just predicting the next token", take a look at this: https://thegradient.pub/othello/ https://thegradient.pub/othello/ As to how it ends up this way, tha
by int_19h 4y ago
Wrt "just predicting the next token", take a look at this: https://thegradient.pub/othello/ https://thegradient.pub/othello/
As to how it ends up this way, that's an interesting question, but it kinda makes sense in a broad way that if you try to "lossily compress" a lot of interrelated data, at some point the only way to continue doing so is by inferring the underlying concepts - i.e. building an internal model of whatever the inputs describe. And, notably, we had to make the models large enough before results indicative of that manifested, regardless of the amount of text thrown at it during training.