4 ms·
> A tokenizer might encode “once upon a time” as “once,” “upon,” “a,” “time,” for example, while encoding “once upon a ” (which has a trailing whitespace) as “o
by kibwen 2y ago
> A tokenizer might encode “once upon a time” as “once,” “upon,” “a,” “time,” for example, while encoding “once upon a ” (which has a trailing whitespace) as “once,” “upon,” “a,” ” .” Depending on how a model is prompted — with “once upon a” or “once upon a ,” — the results may be completely different, because the model doesn’t understand (as a person would) that the meaning is the same.
TechCrunch, I respect that you have a style guide vis a vis punctuation and quotation marks, but please understand when it's appropriate to break the rules. :P
- jazzyjackson 2y agostymied by their own tokenization !
- Terr_ 2y agoWhen this problem comes up in code-related documentation at work, I often fall back onto either a distinct typeface or a background-color shift. That way it's clearer when I'm referring to a literal code-string versus quote marks that are part of the prose.
- RobRivera 2y agoI'm glad I wasnt the only one