6 ms·

Wouldn’t "visually identical” tokens still be valid utf8?

• wolverine876 3y ago
  It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
  • LoganDark 3y ago
    There is no such thing as "UTF-8 that is invalid for my application". Any properly encoded sequence of UTF-8 codepoints is valid UTF-8. Your application may have constraints on the codepoints that it should accept, but that does not suddenly make them improperly encoded. Using visually-similar lookalikes does not make a UTF-8 string invalid.
    • wolverine876 3y ago
      > There is no such thing as "UTF-8 that is invalid for my application". In fact, yes there is! Your application may vary. :)
      • LoganDark 3y ago
        UTF-8 doesn't vary, it's just an encoding.