4 ms·

> Clear win. I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss. > A code p

by Avernar 10y ago

> Clear win. I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss. > A code point string has other niceties, like being indexed by simple integers. Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that. > Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.

paulddraper 10y ago
His point was that the transfer format should be conceptually independent of the processing. Obviously it has to be encoded in RAM in some way, but the programmer doesn't need to worry about the memory layout.
- Avernar 10y ago
  Why? Why should it be conceptually different if it's easy to work with the encoded form? Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice. While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.
  - paulddraper 10y ago
    Think of JSON. The programmer shouldn't have to know that a newline character is written as \n in a JSON string. The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3. 99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.
    - Avernar 10y ago
      Bad example. If you want to embed that string in your code you have to type those 6 characters anyways. A better example is if you want to find a newline in a string. If you do a find it in a UTF-16 string it may be position 8 and a find in UTF-8 may be position 12. Does it matter what the actual number is? NO. You just pass it to the next function or whatever.

by Avernar 10y ago

paulddraper 10y ago

His point was that the transfer format should be conceptually independent of the processing. Obviously it has to be encoded in RAM in some way, but the programmer doesn't need to worry about the memory layout.

Avernar 10y ago
Why? Why should it be conceptually different if it's easy to work with the encoded form? Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice. While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.
- paulddraper 10y ago
  Think of JSON. The programmer shouldn't have to know that a newline character is written as \n in a JSON string. The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3. 99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.
  - Avernar 10y ago
    Bad example. If you want to embed that string in your code you have to type those 6 characters anyways. A better example is if you want to find a newline in a string. If you do a find it in a UTF-16 string it may be position 8 and a find in UTF-8 may be position 12. Does it matter what the actual number is? NO. You just pass it to the next function or whatever.