6 ms·

I like Rust’s approach to UTF-8: strings are rigidly UTF-8, so the general goal is to make encoding and decoding things that you sort out at the I/O boundary, a

I like Rust’s approach to UTF-8: strings are rigidly UTF-8, so the general goal is to make encoding and decoding things that you sort out at the I/O boundary, and then inside the library you can have sanity.

ncmncm 5y ago
Rust deserves kudos for inventing the term "WTF-8", describing e.g. directory entries -- maybe supposed to be UTF-8, but obliged to allow invalid sequences.
- gfody 5y ago
  isn't wtf8 'double utf8' ie the typical utf8 escape encoded in utf8 Ãƒetc? wobble transform 8bit seems like an unnecessary hijacking of a well labeled error state
  - chrismorgan 5y ago
    https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which I think you’ve found). People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.
    - gfody 5y ago
      til, also found this where he apologizes for hijacking it https://news.ycombinator.com/item?id=9613971 https://news.ycombinator.com/item?id=9613971
lmm 5y ago
You cannot have sanity while handling international text as solely UTF-8 bytesequences (or any other encoding that treats sequences of unicode codepoints as always equivalent). Sooner or later you will have to deal with text that contains both Chinese and Japanese characters, so you will need a richer representation.
- chrismorgan 5y ago
  I presume you’re talking about Han unification? It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition.
  - lmm 5y ago
    > It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition. Given that context, I don't see that UTF-8 actually helps much. Your fundamental structure has to look something like a rope of (byte sequence+annotation) entries. With that structure, using different encodings for different segments doesn't make things noticeably worse.
    - TorKlingberg 5y ago
      I'd say having your Chinese and Japanese text in different encodings would make it worse. In a markup language the annotations can be inline. HTML has e.g. <span lang="ja">, which seems to work well enough.
      - AnIdiotOnTheNet 5y ago
        Someday programmers will have learned their lesson about in-band signaling, but apparently it won't be today.
      - lmm 5y ago
        That seems to have all the same problems as having parts of the document in different encodings - e.g. if you move a section of text from one part of the document to another it will suddenly change behaviour.
- 5y ago
arthur2e5 5y ago
Not a fan of the tone, but ngl seven-part implementation is pretty nice. Kind of wanting a Rust version of it.

ncmncm 5y ago

Rust deserves kudos for inventing the term "WTF-8", describing e.g. directory entries -- maybe supposed to be UTF-8, but obliged to allow invalid sequences.

gfody 5y ago
isn't wtf8 'double utf8' ie the typical utf8 escape encoded in utf8 Ãƒetc? wobble transform 8bit seems like an unnecessary hijacking of a well labeled error state
- chrismorgan 5y ago
  https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which I think you’ve found). People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.
  - gfody 5y ago
    til, also found this where he apologizes for hijacking it https://news.ycombinator.com/item?id=9613971 https://news.ycombinator.com/item?id=9613971

lmm 5y ago

You cannot have sanity while handling international text as solely UTF-8 bytesequences (or any other encoding that treats sequences of unicode codepoints as always equivalent). Sooner or later you will have to deal with text that contains both Chinese and Japanese characters, so you will need a richer representation.

chrismorgan 5y ago
I presume you’re talking about Han unification? It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition.
- lmm 5y ago
  > It’s true that sometimes you may need language annotations, which will sometimes also need to be applied to substrings. I don’t think that invalidates my claim that rigid UTF-8 allows you to have sanity, though I will tweak it to state that using UTF-8 is a necessary but not always sufficient condition. Given that context, I don't see that UTF-8 actually helps much. Your fundamental structure has to look something like a rope of (byte sequence+annotation) entries. With that structure, using different encodings for different segments doesn't make things noticeably worse.
  - TorKlingberg 5y ago
    I'd say having your Chinese and Japanese text in different encodings would make it worse. In a markup language the annotations can be inline. HTML has e.g. <span lang="ja">, which seems to work well enough.
    - AnIdiotOnTheNet 5y ago
      Someday programmers will have learned their lesson about in-band signaling, but apparently it won't be today.
    - lmm 5y ago
      That seems to have all the same problems as having parts of the document in different encodings - e.g. if you move a section of text from one part of the document to another it will suddenly change behaviour.
5y ago

arthur2e5 5y ago

Not a fan of the tone, but ngl seven-part implementation is pretty nice. Kind of wanting a Rust version of it.