3 ms·

That's why I said: > > this is either unnecessary or insufficient. There are certainly plenty of circumstances where you don't want to allow non-canonical rep

by a1369209993 6y ago

That's why I said: > > this is either unnecessary or insufficient. There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch.

kazinator 6y ago
That seems a bit like saying that there are more ways to die than just not wearing a seatbelt in an vehicular collision, so why bother with seatbelts.
- a1369209993 6y ago
  > so why bother with seatbelts. No, it is specifically not like that. It's like saying that, in any situation where you need a seatbelt, you also need airbags and a lack of ready-to-become-shrapnel sharp objects scattered around the inside of the vehicle.
  - kazinator 6y ago
    That version just strikes out at a "seatbelts are all you need" strawman argument which nobody made, and adds nothing to the seatbelt topic. For instance, even if a browser validates all incoming UTF-8 from all sources and rejects bad inputs, users can still be duped with URLs that use funny Unicode characters to look legit. But that concern is simply not in the scope of UTF-8 validation and its requirements. Whereas I might want a low-level I/O library to reject overlong UTF-8 forms at the stream level, I probably would not want it deal with Unicode ambiguity issues at that level. A: "I have some code to to fast UTF-8 Validation" B: "That's nice, but you need to restrict more than just UTF-8 to canonical form in order to catch all attacks." B is just signaling that they know a lot about the larger issue from various angles.
temac 6y ago
Overlong encoding is not just non-canonical UTF-8. It is actually forbidden; it is not UTF-8. A decoder without checks can as well not return the codepoints you expect at all (that would be curious but if you try to heavily optimize one, maybe you can end-up with a result that does that). So at the point where you do UTF-8 -> codepoint decoding, it is perfectly reasonable to... validate UTF-8 -> codepoint decoding? (or do a safe decoding that copes with invalid things) Yes you might need to do other kind of validations after that, maybe (or not) some related to Unicode, maybe even some completely unrelated to Unicode and unique to your application, but so what? That does not make UTF-8 encoding validation useless.
masklinn 6y ago
You also said: > there's no legitimate difference between a2 and a3 Which could not be less true. Decomposition and overlong encoding have nothing whatsoever in common. > There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch. This has nothing to do with overlong encoding. Overlong encoding is nothing like decomposition. Composition and decomposition are different valid sequences of codepoints for the same grapheme cluster. Overlong encoding is the invalid non-minimal encoding of a single codepoint. It's not just a difference of opinion as to the convenience of precomposed codepoints.

kazinator 6y ago

That seems a bit like saying that there are more ways to die than just not wearing a seatbelt in an vehicular collision, so why bother with seatbelts.

a1369209993 6y ago
> so why bother with seatbelts. No, it is specifically not like that. It's like saying that, in any situation where you need a seatbelt, you also need airbags and a lack of ready-to-become-shrapnel sharp objects scattered around the inside of the vehicle.
- kazinator 6y ago
  That version just strikes out at a "seatbelts are all you need" strawman argument which nobody made, and adds nothing to the seatbelt topic. For instance, even if a browser validates all incoming UTF-8 from all sources and rejects bad inputs, users can still be duped with URLs that use funny Unicode characters to look legit. But that concern is simply not in the scope of UTF-8 validation and its requirements. Whereas I might want a low-level I/O library to reject overlong UTF-8 forms at the stream level, I probably would not want it deal with Unicode ambiguity issues at that level. A: "I have some code to to fast UTF-8 Validation" B: "That's nice, but you need to restrict more than just UTF-8 to canonical form in order to catch all attacks." B is just signaling that they know a lot about the larger issue from various angles.

temac 6y ago

Overlong encoding is not just non-canonical UTF-8. It is actually forbidden; it is not UTF-8. A decoder without checks can as well not return the codepoints you expect at all (that would be curious but if you try to heavily optimize one, maybe you can end-up with a result that does that). So at the point where you do UTF-8 -> codepoint decoding, it is perfectly reasonable to... validate UTF-8 -> codepoint decoding? (or do a safe decoding that copes with invalid things) Yes you might need to do other kind of validations after that, maybe (or not) some related to Unicode, maybe even some completely unrelated to Unicode and unique to your application, but so what? That does not make UTF-8 encoding validation useless.

masklinn 6y ago

You also said: > there's no legitimate difference between a2 and a3 Which could not be less true. Decomposition and overlong encoding have nothing whatsoever in common. > There are certainly plenty of circumstances where you don't want to allow non-canonical representations; my point was that there are more non-canonical representations than just the ones that UTF-8-level canonicality checks catch. This has nothing to do with overlong encoding. Overlong encoding is nothing like decomposition. Composition and decomposition are different valid sequences of codepoints for the same grapheme cluster. Overlong encoding is the invalid non-minimal encoding of a single codepoint. It's not just a difference of opinion as to the convenience of precomposed codepoints.