4 ms·
This article uses the word 'rune' extensively. From the context I assume it means 'lexem' or 'token' (i.e. the unit the lexer/scanner produces and feeds to pars
by sesm 4y ago
This article uses the word 'rune' extensively. From the context I assume it means 'lexem' or 'token' (i.e. the unit the lexer/scanner produces and feeds to parser). But then the article uses the word 'token' to mean the output of a parser ('keyword token'), while the usual terminology is that parser output is called a 'parse tree'.
So, in this terminology, the parser consumes 'runes' and outputs 'tokens', while the usual terminology is that parser consumes 'tokens'/'lexems' and outputs 'parse tree'.
- pgwhalen 4y agoRune is a type alias in go, which more or less maps to the more common words “character” or “code point”. https://go.dev/blog/strings https://go.dev/blog/strings
- sesm 4y agoThanks! I don't know Go, so this terminology was surprising to me.
- sjansen 4y agoIn Go, `rune` is an alias for `int32` and is used to indicate the value is a Unicode "code point". For characters in the ASCII range, that means it's just a character encoded using more bits. If you need to worry about the full Unicode range then it's important to understand Unicode Normalization Forms. https://go.dev/blog/strings https://go.dev/blog/strings https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence