4 ms·

The libraries are not good. Support for the libraries in runtimes varies wildly and has drastic performance implications: unicode has variable width characters,

The libraries are not good. Support for the libraries in runtimes varies wildly and has drastic performance implications: unicode has variable width characters, so sometimes character array indexes are no longer O(1)! Unicode is not a good system. Yes, we can struggle with it but it will always be a struggle because of the nature of its design. A major omission is a way to indicate the encoding of a particular byte array. You open a file on disk and read its characters -- what encoding is it in? What encoding is used for the various buffers embedded in dozens of historic file formats for audio, video, archives, etc? Developers are forced to guess and hack. The results are about as expected, given the quality of the underlying interfaces.

dontlaugh 4y ago
Why would you index into a "character array"? Also, what is a "character" to you?
- throwaway09223 4y ago
  > Why would you index into a "character array"? Obviously, because you need to examine characters. Here are a couple examples: * You're implementing any software that deals with rfc6531 compliant email. You'll need to parse strings as unicode to understand where to route mail, for example to understand 你好@example.com * You're implementing a database, or any software with a search function which indexes unicode fields. You'll have to understand the logical character structure of the data, or your searches won't match properly. > Also, what is a "character" to you? Characters are defined by the unicode specification. In UTF8 they can be represented by multi-byte sequences. Using the example of 你好@example.com, the @ is the seventh byte position, but the third character position. You cannot implement software that handles internationalized emails without parsing and understanding strings like this.
  - tialaramex 4y ago
    > Characters are defined by the unicode specification. Unicode's glossary offers four definitions for this word, which ought to be a strong hint that this is not a technical term but instead a vague idea people have which doesn't align very well to the technical problem.
    - throwaway09223 4y ago
      As I said initially: The interface (including the underlying unicode specification) is not good. This is precisely my point. Internationalization is building on a foundation of sand.
  - dontlaugh 4y ago
    That's not how you'd parse email. In fact, the bit before @ is encoded differently from the bit after @. You're much more likely to iterate over elements in a string than index. You might iterate over code points, abstract characters, glyphs, grapheme clusters, etc. You might do it before or after a certain kind of normalisation. Since there are many ways to split a string (depending on what you're doing), it won't be possible to make all of them have O(1) indexing.
    - throwaway09223 4y ago
      What do you mean "that's not how you'd parse email?" I didn't describe an algorithm. Why don't you explain how you would parse the above email address without understanding the unicode encoding?
      - colejohnson66 4y ago
        If you're asking how to get everything after the '@' symbol, you can use knowledge of UTF-8 (assuming your byte array is UTF-8) to just say `FindIndexOfByte(utf8Buffer, (byte)'@') + 1`. But treat everything else as an opaque blob. The URL portion of that email address, as far as you should be concerned, it just the bytes after the '@' symbol. If you want to verify if it's a valid URL, you should use a URL library, not some hack that assumes a TLD is 3+ characters. However, even that naive method of a byte search is wrong. According to RFC 2822, '@' symbol is actually allowed on the left hand side if escaped. You should be using an email parsing library that implements the spec properly. Naming the smallest unit of information in C a 'char' was a terrible mistake made decades ago. A character has never been a single byte in many countries. Shift-JIS, for example, can be one or two bytes, and it's two and a half decades old. Even in UTF-16 (used by Java and C#), a 'char' isn't actually a character, but a codepoint. One that could even be an unpaired surrogate. Really, you should never be indexing into the backing byte/char array unless you know what you're doing. If you think you need to, you should be asking yourself, "what is it I really need to do?" Because you, as a programmer dealing with internationalization, must not assume English/Western ideas about language. For example, in abugidas[0], a single word can be a whole sentence. For a more common language, Japanese typically doesn't use spaces because they're not needed when writing kanji. But that's not foolproof because hiragana and katakana aren't for words, but sounds. How do you do a word count in those languages? You don't. [0]: https://en.wikipedia.org/wiki/Abugida https://en.wikipedia.org/wiki/Abugida
  - withinboredom 4y ago
    Why the hell do you need to route the email unless you’re working on an email server? Send the email and have the user validate it. If they validate it, it’s a valid email. Don’t be clever, be smart. Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken or you’re writing an implementation from scratch. In that case, treat it as an opaque blob.
    - throwaway09223 4y ago
      > unless you’re working on an email server? We are talking about email software, inclusive of servers, clients, plugins, etc. It's literally the example I'm offering (but there are neigh infinite others) > Send the email and have the user validate it. I But I haven't proposed a problem of validating an email. Besides, even if the problem is validating an email this is not necessarily a workable answer. > Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken A lot of the arguments supporting unicode seem to boil down to acknowledging that all these things are broken, yet disregarding them because the problems exist at a more technical level than most of HN commenters are familiar with. You might not write this kind of software for a living, but some of us do and we know that these interfaces are not good.
tialaramex 4y ago
> what encoding is it in? UTF-8. Unless you've got clear directions that it needs to be something else, the correct choice is always UTF-8. Stop emitting things in other encodings, "Doctor it hurts when I stab myself with this knife". Yes it does, so don't.
- throwaway09223 4y ago
  That's a great guess, but my point is that it's a guess. It will sometimes be wrong, and when the guess is wrong you will have articles like the above written complaining about the fallout. These types of errors are guaranteed specifically because of the policy of guessing and loose heuristics you're describing.
  - GoblinSlayer 4y ago
    Unicode fixes it. It's why legacy encodings are unworkable.
    - throwaway09223 4y ago
      Unicode absolutely does not fix the issue of not knowing the encoding of a legacy byte buffer with text in it.
cedilla 4y ago
> unicode has variable width characters, so sometimes character array indexes are no longer O(1)! > Unicode is not a good system. Unicode is excellent. Your gripe is with human languages and their scripts. It wasn't the Unicode consortium who decided that some scripts have oddities like single glyphs that can encode anything between a single sound and a whole sentence, or oddities like different 'cases' where glyphs change form just because they happen to be at the beginning of a sentence or proper noun

dontlaugh 4y ago

Why would you index into a "character array"? Also, what is a "character" to you?

throwaway09223 4y ago
> Why would you index into a "character array"? Obviously, because you need to examine characters. Here are a couple examples: * You're implementing any software that deals with rfc6531 compliant email. You'll need to parse strings as unicode to understand where to route mail, for example to understand 你好@example.com * You're implementing a database, or any software with a search function which indexes unicode fields. You'll have to understand the logical character structure of the data, or your searches won't match properly. > Also, what is a "character" to you? Characters are defined by the unicode specification. In UTF8 they can be represented by multi-byte sequences. Using the example of 你好@example.com, the @ is the seventh byte position, but the third character position. You cannot implement software that handles internationalized emails without parsing and understanding strings like this.
- tialaramex 4y ago
  > Characters are defined by the unicode specification. Unicode's glossary offers four definitions for this word, which ought to be a strong hint that this is not a technical term but instead a vague idea people have which doesn't align very well to the technical problem.
  - throwaway09223 4y ago
    As I said initially: The interface (including the underlying unicode specification) is not good. This is precisely my point. Internationalization is building on a foundation of sand.
- dontlaugh 4y ago
  That's not how you'd parse email. In fact, the bit before @ is encoded differently from the bit after @. You're much more likely to iterate over elements in a string than index. You might iterate over code points, abstract characters, glyphs, grapheme clusters, etc. You might do it before or after a certain kind of normalisation. Since there are many ways to split a string (depending on what you're doing), it won't be possible to make all of them have O(1) indexing.
  - throwaway09223 4y ago
    What do you mean "that's not how you'd parse email?" I didn't describe an algorithm. Why don't you explain how you would parse the above email address without understanding the unicode encoding?
    - colejohnson66 4y ago
      If you're asking how to get everything after the '@' symbol, you can use knowledge of UTF-8 (assuming your byte array is UTF-8) to just say `FindIndexOfByte(utf8Buffer, (byte)'@') + 1`. But treat everything else as an opaque blob. The URL portion of that email address, as far as you should be concerned, it just the bytes after the '@' symbol. If you want to verify if it's a valid URL, you should use a URL library, not some hack that assumes a TLD is 3+ characters. However, even that naive method of a byte search is wrong. According to RFC 2822, '@' symbol is actually allowed on the left hand side if escaped. You should be using an email parsing library that implements the spec properly. Naming the smallest unit of information in C a 'char' was a terrible mistake made decades ago. A character has never been a single byte in many countries. Shift-JIS, for example, can be one or two bytes, and it's two and a half decades old. Even in UTF-16 (used by Java and C#), a 'char' isn't actually a character, but a codepoint. One that could even be an unpaired surrogate. Really, you should never be indexing into the backing byte/char array unless you know what you're doing. If you think you need to, you should be asking yourself, "what is it I really need to do?" Because you, as a programmer dealing with internationalization, must not assume English/Western ideas about language. For example, in abugidas[0], a single word can be a whole sentence. For a more common language, Japanese typically doesn't use spaces because they're not needed when writing kanji. But that's not foolproof because hiragana and katakana aren't for words, but sounds. How do you do a word count in those languages? You don't. [0]: https://en.wikipedia.org/wiki/Abugida https://en.wikipedia.org/wiki/Abugida
- withinboredom 4y ago
  Why the hell do you need to route the email unless you’re working on an email server? Send the email and have the user validate it. If they validate it, it’s a valid email. Don’t be clever, be smart. Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken or you’re writing an implementation from scratch. In that case, treat it as an opaque blob.
  - throwaway09223 4y ago
    > unless you’re working on an email server? We are talking about email software, inclusive of servers, clients, plugins, etc. It's literally the example I'm offering (but there are neigh infinite others) > Send the email and have the user validate it. I But I haven't proposed a problem of validating an email. Besides, even if the problem is validating an email this is not necessarily a workable answer. > Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken A lot of the arguments supporting unicode seem to boil down to acknowledging that all these things are broken, yet disregarding them because the problems exist at a more technical level than most of HN commenters are familiar with. You might not write this kind of software for a living, but some of us do and we know that these interfaces are not good.

tialaramex 4y ago

> what encoding is it in? UTF-8. Unless you've got clear directions that it needs to be something else, the correct choice is always UTF-8. Stop emitting things in other encodings, "Doctor it hurts when I stab myself with this knife". Yes it does, so don't.

throwaway09223 4y ago
That's a great guess, but my point is that it's a guess. It will sometimes be wrong, and when the guess is wrong you will have articles like the above written complaining about the fallout. These types of errors are guaranteed specifically because of the policy of guessing and loose heuristics you're describing.
- GoblinSlayer 4y ago
  Unicode fixes it. It's why legacy encodings are unworkable.
  - throwaway09223 4y ago
    Unicode absolutely does not fix the issue of not knowing the encoding of a legacy byte buffer with text in it.

cedilla 4y ago

> unicode has variable width characters, so sometimes character array indexes are no longer O(1)! > Unicode is not a good system. Unicode is excellent. Your gripe is with human languages and their scripts. It wasn't the Unicode consortium who decided that some scripts have oddities like single glyphs that can encode anything between a single sound and a whole sentence, or oddities like different 'cases' where glyphs change form just because they happen to be at the beginning of a sentence or proper noun