3 ms·

Your UTF-8 decoder is identical to just "emit a REPLACEMENT CHARACTER for every bogus byte". Also, the use of surrogate pairs mean you actually don't have a UT

Your UTF-8 decoder is identical to just "emit a REPLACEMENT CHARACTER for every bogus byte". Also, the use of surrogate pairs mean you actually don't have a UTF-8 decoder at all, you just have something that's similar but produces a sequence of potentially-invalid unicode scalars. > we also need to ensure that when some U+DCXX occurs by means of a valid UTF-8 pattern It can't. Trying to encode surrogate pair codepoints in UTF-8 is strictly invalid.

Dylan16807 9y ago
> It can't. Why did you cut off "we nevertheless treat it as invalid."? You're misreading that line, and actually in agreement on what to do. It's the bit pattern that is valid, which is how you can parse the code point and see that the code point is invalid for UTF-8.
- eridius 9y ago
  I cut you off because you basically said "and if we see this thing that is defined as invalid, we treat it as invalid". Which is something that all UTF-8 parsers do, so there's no point in calling it out like this is special behavior.
  - Dylan16807 9y ago
    That's not me. The point is that this variant parser accepts many things that are normally errors and turns them into faux-surrogates, so it's worth restating that it rejects surrogates in the source file.
    - eridius 9y ago
      My apologies. In any case, here's what OP said: > Well, that's not the full story: to have this transparency property, we also need to ensure that when some U+DCXX occurs by means of a valid UTF-8 pattern, we nevertheless treat it as invalid. I.e. there is a rule that if the UTF-8 decode works, but a U+DCXX code-point emerges, then we retreat to the start and drop a byte as a replacement character, as if a bad code had been seen. But this whole paragraph is wrongheaded. A U+DCXX codepoint cannot occur as a result of a UTF-8 decode, because it is defined as invalid. Even just that last bit there, "as if a bad code had been seen"... a bad code was seen! This paragraph makes me question whether OP actually understands UTF-8 decoding at all.
      - Dylan16807 9y ago
        I think the meaning is clear. They're talking about part of the UTF-8 decoding process, the part that actually decodes leading/trailing bytes into arbitrary 21 bit numbers. What term would you use for that?
        eridius 9y ago
        Well, I wouldn't even bring it up, because the UTF-8 decoding process by definition cannot produce code points in the surrogate pair range, as those are illegal to encode in UTF-8. But if I must, I might talk about the "bit pattern" and the integral value that results from interpreting it. I certainly wouldn't talk about code points resulting from a UTF-8 decode.
        eridius 9y ago
        Somebody is apparently upset about what I said. Why?

Dylan16807 9y ago

> It can't. Why did you cut off "we nevertheless treat it as invalid."? You're misreading that line, and actually in agreement on what to do. It's the bit pattern that is valid, which is how you can parse the code point and see that the code point is invalid for UTF-8.

eridius 9y ago
I cut you off because you basically said "and if we see this thing that is defined as invalid, we treat it as invalid". Which is something that all UTF-8 parsers do, so there's no point in calling it out like this is special behavior.
- Dylan16807 9y ago
  That's not me. The point is that this variant parser accepts many things that are normally errors and turns them into faux-surrogates, so it's worth restating that it rejects surrogates in the source file.
  - eridius 9y ago
    My apologies. In any case, here's what OP said: > Well, that's not the full story: to have this transparency property, we also need to ensure that when some U+DCXX occurs by means of a valid UTF-8 pattern, we nevertheless treat it as invalid. I.e. there is a rule that if the UTF-8 decode works, but a U+DCXX code-point emerges, then we retreat to the start and drop a byte as a replacement character, as if a bad code had been seen. But this whole paragraph is wrongheaded. A U+DCXX codepoint cannot occur as a result of a UTF-8 decode, because it is defined as invalid. Even just that last bit there, "as if a bad code had been seen"... a bad code was seen! This paragraph makes me question whether OP actually understands UTF-8 decoding at all.
    - Dylan16807 9y ago
      I think the meaning is clear. They're talking about part of the UTF-8 decoding process, the part that actually decodes leading/trailing bytes into arbitrary 21 bit numbers. What term would you use for that?
      - eridius 9y ago
        Well, I wouldn't even bring it up, because the UTF-8 decoding process by definition cannot produce code points in the surrogate pair range, as those are illegal to encode in UTF-8. But if I must, I might talk about the "bit pattern" and the integral value that results from interpreting it. I certainly wouldn't talk about code points resulting from a UTF-8 decode.
        eridius 9y ago
        Somebody is apparently upset about what I said. Why?