3 ms·

That's precisely what I meant with your UCS-2 comment. What do you mean by variable width? the 4 bit header would be limited to the first code unit, the follow

That's precisely what I meant with your UCS-2 comment. What do you mean by variable width? the 4 bit header would be limited to the first code unit, the following ones would just be pure codepoint values. and that's a good point, I as picturing it as the top 4 bits that describe the format, so the header's byte order would be field, but the actual codepoint value would be encoded with whatever byte and bit order the header indicated.

Dylan16807 8y ago
Oh, so it's a BOM? Why integrate it into the code unit if you're only going to have one of them?
- bumblebritches5 8y ago
  > Oh, so it's a BOM? Kinda? It isn't limited to just the byte order tho, but I can see it. > Why integrate it into the code unit if you're only going to have one of them? There would be multiple code units, just 1 of the mini headers/BOM. With that comment I was trying to contrast it to UTF-8 which has those 2 leading bits of the continuation code units set to `0b10`, it wouldn't have that.
  - Dylan16807 8y ago
    Oh, the first code unit of each code point, not the first code unit of the text. That's how I had originally understood you until I got confused by the talk of byte order. I have to say I really don't see the value of being byte-order flexible, but always requiring the header be in the "first" byte, because that makes it so big-endian and little-endian formats are packed completely differently. 2 and 4 byte code units will be misaligned all the time too, might as well just use a series of bytes in fixed order. > What do you mean by variable width? Make the first byte be like UTF-8 (so a prefix of 0, 10, 110, or 1110), and the following bytes be raw data. It compacts a lot better than a fixed-length header, with roughly the same level of complexity. But now that I think about it a continuation bit is probably the best option here because it regains some ability to self-synchronize.

Dylan16807 8y ago

Oh, so it's a BOM? Why integrate it into the code unit if you're only going to have one of them?

bumblebritches5 8y ago
> Oh, so it's a BOM? Kinda? It isn't limited to just the byte order tho, but I can see it. > Why integrate it into the code unit if you're only going to have one of them? There would be multiple code units, just 1 of the mini headers/BOM. With that comment I was trying to contrast it to UTF-8 which has those 2 leading bits of the continuation code units set to `0b10`, it wouldn't have that.
- Dylan16807 8y ago
  Oh, the first code unit of each code point, not the first code unit of the text. That's how I had originally understood you until I got confused by the talk of byte order. I have to say I really don't see the value of being byte-order flexible, but always requiring the header be in the "first" byte, because that makes it so big-endian and little-endian formats are packed completely differently. 2 and 4 byte code units will be misaligned all the time too, might as well just use a series of bytes in fixed order. > What do you mean by variable width? Make the first byte be like UTF-8 (so a prefix of 0, 10, 110, or 1110), and the following bytes be raw data. It compacts a lot better than a fixed-length header, with roughly the same level of complexity. But now that I think about it a continuation bit is probably the best option here because it regains some ability to self-synchronize.