3 ms·

As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.

orangepanda 3y ago
Wouldn’t "visually identical” tokens still be valid utf8?
- wolverine876 3y ago
  It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
  - LoganDark 3y ago
    There is no such thing as "UTF-8 that is invalid for my application". Any properly encoded sequence of UTF-8 codepoints is valid UTF-8. Your application may have constraints on the codepoints that it should accept, but that does not suddenly make them improperly encoded. Using visually-similar lookalikes does not make a UTF-8 string invalid.
    - wolverine876 3y ago
      > There is no such thing as "UTF-8 that is invalid for my application". In fact, yes there is! Your application may vary. :)
      - LoganDark 3y ago
        UTF-8 doesn't vary, it's just an encoding.
LoganDark 3y ago
That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string. [0]: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

orangepanda 3y ago

Wouldn’t "visually identical” tokens still be valid utf8?

wolverine876 3y ago
It depends on the meaning of 'valid UTF-8'. They could be 'UTF-8 that is invalid for my application', but still be 'strings that are valid under the UTF-8 specification'.
- LoganDark 3y ago
  There is no such thing as "UTF-8 that is invalid for my application". Any properly encoded sequence of UTF-8 codepoints is valid UTF-8. Your application may have constraints on the codepoints that it should accept, but that does not suddenly make them improperly encoded. Using visually-similar lookalikes does not make a UTF-8 string invalid.
  - wolverine876 3y ago
    > There is no such thing as "UTF-8 that is invalid for my application". In fact, yes there is! Your application may vary. :)
    - LoganDark 3y ago
      UTF-8 doesn't vary, it's just an encoding.

LoganDark 3y ago

That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string. [0]: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/