4 ms·
It means the word "the" as part of instructions and the word "the" as part of data would be two different tokens
by inigyou 5mo ago
It means the word "the" as part of instructions and the word "the" as part of data would be two different tokens
- danlitt 5mo agoBut tokens are just text! Isn't it all just text? If you're training and you encounter "the", is that an instruction "the" or a data "the"?
- inigyou 5mo agoIf it occurs in the text box for instructions you encode it as an instruction "the" and if it occurs in the text box for data you encode it as a data "the"
- tekne 5mo agoExactly! Think of how an image of a car and a car in front of you may look indistinguishable in 2D -- but due to your 3D vision you know they're not the same thing (but also know the image is of a car, while not literally being a car). Likewise, blue tokens are the image of red tokens.