3 ms·

"Guessing the next token" is an interface, not an implementation. A transformer is a universal function approximator and is capable of learning the function "gi

"Guessing the next token" is an interface, not an implementation. A transformer is a universal function approximator and is capable of learning the function "give the right answer" through this interface. (Also, if you do beam search, it's not "the next token" but "the next tokens".)

dr_dshiv 3y ago
Wow, I’d love to have that explained in more detail — since I hear it all the time. Any source or write up?
- astrange 3y ago
  https://arxiv.org/pdf/2103.05247.pdf https://arxiv.org/pdf/2103.05247.pdf https://openreview.net/pdf?id=gEZrGCozdqR https://openreview.net/pdf?id=gEZrGCozdqR