3 ms·
> If not for that requirement, we would have just had UTF-8 implemented as regular VLQs. While not technically equivalent, synchronized byte stream allows a ba
by lifthrasiir 7y ago
> If not for that requirement, we would have just had UTF-8 implemented as regular VLQs.
While not technically equivalent, synchronized byte stream allows a backward scanning that is beneficial for many Unicode-related algorithms. In fact synchronization is the simplest way to do that.
- kstenerud 7y agoVLQ can also be backward scanned, since the high bit is the continuation bit. Just scan for the next cleared high bit, which marks the end of a glyph.
- lifthrasiir 7y agoAh, good point. I'm still confusing which one is which. I was specifically thinking of UTF-1 that is definitely not synchronized nor backward scannable.
- SamReidHughes 7y agoProvided that you know where the string boundary is, so you don't scan off the front into differently encoded data. So it can't be backward scanned for some miscellaneous encoded number.