3 ms·

> Thus you must validate that the strings you receive are valid UTF-8. Depending on your application, this is either unnecessary or insufficient. Consider a1="

by a1369209993 6y ago

> Thus you must validate that the strings you receive are valid UTF-8. Depending on your application, this is either unnecessary or insufficient. Consider a1="\xC3\xA4", a2="\x61\xCC\x88", a3="\xE0\x83\xA4". Assuming you're using a1 as your canonical representation of "ä", there's no legitimate difference between a2 and a3 despite the former being 'valid' UTF-8 and the latter not. (There are decent reasons why you might use a2 as your canonical representation instead, whereas the reasons for using a3 are satyrically flimsy (ironically they'd better support using a4="\xF0\x80\x83\xA4"), but that would just leave no legitimate difference between a1 and a3.)

masklinn 6y ago
There are rather good reasons to not allow overlong encoding. For instance it allows smuggling data through unicode-unaware filters into unicode-aware applications, and while you should also catch the issue on the other side of the filter, overlong encoding allows trivially bypassing one line of defence for no reason. Such filters usually don't usually care much about precomposition or decomposition.
- a1369209993 6y ago
  That's why I said: > > this is either unnecessary or insufficient. There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch.
  - kazinator 6y ago
    That seems a bit like saying that there are more ways to die than just not wearing a seatbelt in an vehicular collision, so why bother with seatbelts.
    - a1369209993 6y ago
      > so why bother with seatbelts. No, it is specifically not like that. It's like saying that, in any situation where you need a seatbelt, you also need airbags and a lack of ready-to-become-shrapnel sharp objects scattered around the inside of the vehicle.
      - kazinator 6y ago
        That version just strikes out at a "seatbelts are all you need" strawman argument which nobody made, and adds nothing to the seatbelt topic. For instance, even if a browser validates all incoming UTF-8 from all sources and rejects bad inputs, users can still be duped with URLs that use funny Unicode characters to look legit. But that concern is simply not in the scope of UTF-8 validation and its requirements. Whereas I might want a low-level I/O library to reject overlong UTF-8 forms at the stream level, I probably would not want it deal with Unicode ambiguity issues at that level. A: "I have some code to to fast UTF-8 Validation" B: "That's nice, but you need to restrict more than just UTF-8 to canonical form in order to catch all attacks." B is just signaling that they know a lot about the larger issue from various angles.
  - temac 6y ago
    Overlong encoding is not just non-canonical UTF-8. It is actually forbidden; it is not UTF-8. A decoder without checks can as well not return the codepoints you expect at all (that would be curious but if you try to heavily optimize one, maybe you can end-up with a result that does that). So at the point where you do UTF-8 -> codepoint decoding, it is perfectly reasonable to... validate UTF-8 -> codepoint decoding? (or do a safe decoding that copes with invalid things) Yes you might need to do other kind of validations after that, maybe (or not) some related to Unicode, maybe even some completely unrelated to Unicode and unique to your application, but so what? That does not make UTF-8 encoding validation useless.
  - masklinn 6y ago
    You also said: > there's no legitimate difference between a2 and a3 Which could not be less true. Decomposition and overlong encoding have nothing whatsoever in common. > There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch. This has nothing to do with overlong encoding. Overlong encoding is nothing like decomposition. Composition and decomposition are different valid sequences of codepoints for the same grapheme cluster. Overlong encoding is the invalid non-minimal encoding of a single codepoint. It's not just a difference of opinion as to the convenience of precomposed codepoints.

masklinn 6y ago

There are rather good reasons to not allow overlong encoding. For instance it allows smuggling data through unicode-unaware filters into unicode-aware applications, and while you should also catch the issue on the other side of the filter, overlong encoding allows trivially bypassing one line of defence for no reason. Such filters usually don't usually care much about precomposition or decomposition.

a1369209993 6y ago
That's why I said: > > this is either unnecessary or insufficient. There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch.
- kazinator 6y ago
  That seems a bit like saying that there are more ways to die than just not wearing a seatbelt in an vehicular collision, so why bother with seatbelts.
  - a1369209993 6y ago
    > so why bother with seatbelts. No, it is specifically not like that. It's like saying that, in any situation where you need a seatbelt, you also need airbags and a lack of ready-to-become-shrapnel sharp objects scattered around the inside of the vehicle.
    - kazinator 6y ago
      That version just strikes out at a "seatbelts are all you need" strawman argument which nobody made, and adds nothing to the seatbelt topic. For instance, even if a browser validates all incoming UTF-8 from all sources and rejects bad inputs, users can still be duped with URLs that use funny Unicode characters to look legit. But that concern is simply not in the scope of UTF-8 validation and its requirements. Whereas I might want a low-level I/O library to reject overlong UTF-8 forms at the stream level, I probably would not want it deal with Unicode ambiguity issues at that level. A: "I have some code to to fast UTF-8 Validation" B: "That's nice, but you need to restrict more than just UTF-8 to canonical form in order to catch all attacks." B is just signaling that they know a lot about the larger issue from various angles.
- temac 6y ago
  Overlong encoding is not just non-canonical UTF-8. It is actually forbidden; it is not UTF-8. A decoder without checks can as well not return the codepoints you expect at all (that would be curious but if you try to heavily optimize one, maybe you can end-up with a result that does that). So at the point where you do UTF-8 -> codepoint decoding, it is perfectly reasonable to... validate UTF-8 -> codepoint decoding? (or do a safe decoding that copes with invalid things) Yes you might need to do other kind of validations after that, maybe (or not) some related to Unicode, maybe even some completely unrelated to Unicode and unique to your application, but so what? That does not make UTF-8 encoding validation useless.
- masklinn 6y ago
  You also said: > there's no legitimate difference between a2 and a3 Which could not be less true. Decomposition and overlong encoding have nothing whatsoever in common. > There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch. This has nothing to do with overlong encoding. Overlong encoding is nothing like decomposition. Composition and decomposition are different valid sequences of codepoints for the same grapheme cluster. Overlong encoding is the invalid non-minimal encoding of a single codepoint. It's not just a difference of opinion as to the convenience of precomposed codepoints.