The most curious errors I got when testing a Unicode-enabled lexer were not from the lexer itself. These are valid C-style identifiers if you extend those rule

The most curious errors I got when testing a Unicode-enabled lexer were not from the lexer itself. These are valid C-style identifiers if you extend those rules to Unicode, and editors have no problem with them: fish_3 рыбы_3 So are these, despite the combination of left-to-right and right-to-left characters: سمك_٣ (Eastern Arabic numeral for three, note how it appears to come first to an LTR reader.) سمك_3 (Western Arabic numeral for three) Problem is, editors really have trouble dealing with mixed-direction words like this. The caret stops pointing to the correct part of the word, the home and end keys become useless, etc. It's an odd situation where the lexer can handle these edge cases well but I cannot think of how you'd actually get them into the compiler.