2 ms·
https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes is a big part of it and always has been (but note this should only apply to letters/words/phonetics, a
by gwern 3y ago
https://gwern.net/gpt-3#bpes https://gwern.net/gpt-3#bpes is a big part of it and always has been (but note this should only apply to letters/words/phonetics, and the sentence issues should be something else).
- hammock 3y agoThanks for linking that explanation. Everyone has been saying “tokens” which I get but before your comment, the missing piece for me was that these tokens are of arbitrary (and suboptimally long) length as a tradeoff for performance