4 ms·
It's mentioned in the article: "This example simplifies things a little bit because in actuality tokens do not map 1:1 to ASCII characters (a popular token enco
by gattilorenz 3y ago
It's mentioned in the article: "This example simplifies things a little bit because in actuality tokens do not map 1:1 to ASCII characters (a popular token encoding technique is Byte-Pair Encoding which is beyond the scope of this blog post), but the iterative nature of generation is the same regardless of how you tokenize your sequences."
It's an extremization that still is true for character-based models.