2 ms·
From the point of view of the model, there is no text, only a stream of tokens. There is no need for all tokens to map to valid ascii or unicode. You could have
by wongarsu 4y ago
From the point of view of the model, there is no text, only a stream of tokens. There is no need for all tokens to map to valid ascii or unicode. You could have an arbitrary number of tokens for control flow that mean something to the software, but don't have a canonical way to encode in text.