4 ms·
Tokens do not encode semantics.
by measurablefunc 9mo ago
Tokens do not encode semantics.
- WithinReason 9mo agoYou can choose which token to sample based on language semantics. You simply don't sample invalid ones. So the language should be restrictive on what tokens it allows enough that invalid code is impossible.
- SkiFire13 9mo ago> You can choose which token to sample based on language semantics Can you though? > the language should be restrictive on what tokens it allows This is a restriction on the language syntax, not its semantics.