4 ms·

> Now, if characters were objects, their representation would be encapsulated, and nobody would very much affected how many bits are needed. Making characters

> Now, if characters were objects, their representation would be encapsulated, and nobody would very much affected how many bits are needed. Making characters objects doesn't seem like it magics away this issue. You'd need APIs to get and set raw values somewhere, and those APIs would have become the problem.

lmm 7y ago
> Making characters objects doesn't seem like it magics away this issue. You'd need APIs to get and set raw values somewhere, and those APIs would have become the problem. You never need to "get and set raw values". You need APIs to do the things that are sensible to do with characters, such as "test whether it has this property" or "encode as UTF-16 byte sequence", but all those things have much clearer semantics than "access raw value".
- pmontra 7y ago
  I just had to write this code in Ruby to convert a list of integers from an Elixir log file into some text that I could read def to_string(s) s.split(", ").map { |dec| dec.to_i.chr(Encoding::UTF_8) }.join end "100, 8217, 65, 118, 111, 114, 105, 111".to_string => d’Avorio I had to specify the encoding because an unassisted Fixnum#chr works only for ASCII values. However I can imagine that it could use some intelligence to figure out the correct encoding. Probably it's not worth it and it could make mistakes.
  - lmm 7y ago
    Exactly - "turn this value from this specified encoding into a character" is semantically much clearer than "make a character whose internal representation is this integer".
- nullwasamistake 7y ago
  You need raw values for serialization over the wire and to files. And for any kind of compression. And for searching (it's hard to build indexes in Unicode because of uneven symbol widths). You could convert to a different character format but at some point you need raw values. If you want to be able to read text created by something else (text editor, browser, different OS) you have to expose raw access.
  - lmm 7y ago
    > You need raw values for serialization over the wire and to files. And for any kind of compression. You need to encode as bytes with a particular encoding. That's not the same thing as "getting the raw value". > And for searching (it's hard to build indexes in Unicode because of uneven symbol widths). It may be hard but it's important to do it right. If searching for "café" doesn't find "caf◌́e" then you've got a problem. > You could convert to a different character format but at some point you need raw values. If you want to be able to read text created by something else (text editor, browser, different OS) you have to expose raw access. Again, no. You need to be able to decode byte sequences that have particular encodings into strings of characters. That doesn't mean you need to expose the internal representation of those strings/characters.
    - nullwasamistake 7y ago
      > Again, no. You need to be able to decode byte sequences that have particular encodings into strings of characters. You keep reiterating this, but it's not feasible. To avoid exposing the raw values, the language would need to support all possible encodings.
      - lmm 7y ago
        Nonsense. What is it that you can do with a "raw value" that you can't do with the representation in a particular encoding? I mean, if you wanted to import a character that doesn't have a unicode codepoint then you couldn't decode that character from UTF-8 - but if the language is built with no support for non-unicode characters then even if you did have access to the internal representation of a character, that wouldn't help you (e.g. the language's built-in character functions for things like checking the case of the character won't handle a non-unicode character properly).
        nullwasamistake 7y ago
        In Java at least, you can write your own Charset implementation then the language will support it normally. This uses the byte raw access I'm ranting about to work
- nitwit005 7y ago
  Java does have quite nice classes for doing that sort of thing, but those classes need to be able to get and set the raw values somehow. And, of course, users may need to convert their own binary formats to strings (compression and the like). At some point you'll be dealing with raw values.
  - lmm 7y ago
    You need the ability to convert between characters and some standard binary representation of them. But the language doesn't need to use that same binary representation as its internal representation, or expose the internal representation at all.

lmm 7y ago

> Making characters objects doesn't seem like it magics away this issue. You'd need APIs to get and set raw values somewhere, and those APIs would have become the problem. You never need to "get and set raw values". You need APIs to do the things that are sensible to do with characters, such as "test whether it has this property" or "encode as UTF-16 byte sequence", but all those things have much clearer semantics than "access raw value".

pmontra 7y ago
I just had to write this code in Ruby to convert a list of integers from an Elixir log file into some text that I could read def to_string(s) s.split(", ").map { |dec| dec.to_i.chr(Encoding::UTF_8) }.join end "100, 8217, 65, 118, 111, 114, 105, 111".to_string => d’Avorio I had to specify the encoding because an unassisted Fixnum#chr works only for ASCII values. However I can imagine that it could use some intelligence to figure out the correct encoding. Probably it's not worth it and it could make mistakes.
- lmm 7y ago
  Exactly - "turn this value from this specified encoding into a character" is semantically much clearer than "make a character whose internal representation is this integer".
nullwasamistake 7y ago
You need raw values for serialization over the wire and to files. And for any kind of compression. And for searching (it's hard to build indexes in Unicode because of uneven symbol widths). You could convert to a different character format but at some point you need raw values. If you want to be able to read text created by something else (text editor, browser, different OS) you have to expose raw access.
- lmm 7y ago
  > You need raw values for serialization over the wire and to files. And for any kind of compression. You need to encode as bytes with a particular encoding. That's not the same thing as "getting the raw value". > And for searching (it's hard to build indexes in Unicode because of uneven symbol widths). It may be hard but it's important to do it right. If searching for "café" doesn't find "caf◌́e" then you've got a problem. > You could convert to a different character format but at some point you need raw values. If you want to be able to read text created by something else (text editor, browser, different OS) you have to expose raw access. Again, no. You need to be able to decode byte sequences that have particular encodings into strings of characters. That doesn't mean you need to expose the internal representation of those strings/characters.
  - nullwasamistake 7y ago
    > Again, no. You need to be able to decode byte sequences that have particular encodings into strings of characters. You keep reiterating this, but it's not feasible. To avoid exposing the raw values, the language would need to support all possible encodings.
    - lmm 7y ago
      Nonsense. What is it that you can do with a "raw value" that you can't do with the representation in a particular encoding? I mean, if you wanted to import a character that doesn't have a unicode codepoint then you couldn't decode that character from UTF-8 - but if the language is built with no support for non-unicode characters then even if you did have access to the internal representation of a character, that wouldn't help you (e.g. the language's built-in character functions for things like checking the case of the character won't handle a non-unicode character properly).
      - nullwasamistake 7y ago
        In Java at least, you can write your own Charset implementation then the language will support it normally. This uses the byte raw access I'm ranting about to work
nitwit005 7y ago
Java does have quite nice classes for doing that sort of thing, but those classes need to be able to get and set the raw values somehow. And, of course, users may need to convert their own binary formats to strings (compression and the like). At some point you'll be dealing with raw values.
- lmm 7y ago
  You need the ability to convert between characters and some standard binary representation of them. But the language doesn't need to use that same binary representation as its internal representation, or expose the internal representation at all.