4 ms·

One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can

by ekidd 3y ago

One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points). This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type. For another good alternative to this approach, see Ruby.

Animats 3y ago
> separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as s(5)Hello This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems. Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check. Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as (SMILING FACE)(RED HEART) per the spec.
- alcover 3y ago
  Greetings, Professor Falken. (SMILING FACE)(RED HEART)
anonzzzies 3y ago
> "Strings are always UTF-8. Anything else is a bug. Deal with it." That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.

by ekidd 3y ago

Animats 3y ago

> separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as s(5)Hello This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems. Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check. Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as (SMILING FACE)(RED HEART) per the spec.

alcover 3y ago
Greetings, Professor Falken. (SMILING FACE)(RED HEART)

anonzzzies 3y ago

> "Strings are always UTF-8. Anything else is a bug. Deal with it." That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.