2 ms·
Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point. The latter is fiendishly difficult to get righ
by samatman 7y ago
Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point.
The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.
- smasher164 7y agoIt's not though. Replacing invalid byte sequences is not terribly difficult. https://golang.org/src/strings/strings.go?s=15854:15900#L627 https://golang.org/src/strings/strings.go?s=15854:15900#L627.
- samatman 7y agoWe are agreeing, the part I was indicating is difficult is 'displaying gibberish'. Knowing what constitutes a grapheme cluster is detailed and frequently changes.