3 ms·
Presumably, the tokenizer doesn't know anything about the meaning of the tokens it's extracting. It's the parser's job to attach meaning to tokens.
by archivator 15y ago
Presumably, the tokenizer doesn't know anything about the meaning of the tokens it's extracting. It's the parser's job to attach meaning to tokens.
- burgerbrain 15y agoPresumably, the programmers should pick reasonable token names for the sake of other programmers. They're all just numbers to the tokenizer...