4 ms·

How often are stream truncated at the start? In my career, I've seen plenty of end truncation, but start truncation never happens. Or, to be more precise, it on

How often are stream truncated at the start? In my career, I've seen plenty of end truncation, but start truncation never happens. Or, to be more precise, it only happens if previous decoding is already borked. If a previous decoding read too much data, then even UTF-8 is borked. You could be decoding UTF-8 from the bits of any follow-up data. Even for pure text data, if a previous field was over-read (the only plausible way to have start-truncation), then you probably are decoding incorrect data from then on. IOW, this upside is both ludicrously improbable and much more damning to the decoding than simply be able to skip a character.

maxdamantus 5y ago
It happens all the time when you're working with substrings. Under your encoding, "é" would be a substring of "ჩ": 10000011 11101001 U+E9 "é" 10000001 10000011 11101001 U+10E9 "ჩ" This would make it incompatible with many existing processes that already handle text, which was one of the goals of UTF-8.
- supuun 5y ago
  it's nice seeing my native language's character: ჩ
adrianmonk 5y ago
Imagine you plug a dumb terminal into an RS232 port. The computer may have already sent some data. Or it may be in the middle of sending log messages to the console as you plug in. With a code like this, your dumb terminal can be built so it just automatically works. Without it, the terminal may be off by some number of bytes, and you have to have some sort of other (probably manual) synchronization procedure. Similarly, imagine you've dialed in over modem to a remote system. You get some errors or lost text due to noise on the phone line. Or some bytes are lost due to bad RS232 flow control or no flow control. Your terminal emulator would be able to recover. There might also be uses where text is broadcast, such as in TV closed captions, although maybe those have their own framing so you know where strings begin.

maxdamantus 5y ago

It happens all the time when you're working with substrings. Under your encoding, "é" would be a substring of "ჩ": 10000011 11101001 U+E9 "é" 10000001 10000011 11101001 U+10E9 "ჩ" This would make it incompatible with many existing processes that already handle text, which was one of the goals of UTF-8.

supuun 5y ago
it's nice seeing my native language's character: ჩ

adrianmonk 5y ago

Imagine you plug a dumb terminal into an RS232 port. The computer may have already sent some data. Or it may be in the middle of sending log messages to the console as you plug in. With a code like this, your dumb terminal can be built so it just automatically works. Without it, the terminal may be off by some number of bytes, and you have to have some sort of other (probably manual) synchronization procedure. Similarly, imagine you've dialed in over modem to a remote system. You get some errors or lost text due to noise on the phone line. Or some bytes are lost due to bad RS232 flow control or no flow control. Your terminal emulator would be able to recover. There might also be uses where text is broadcast, such as in TV closed captions, although maybe those have their own framing so you know where strings begin.