6 ms·

if you cannot handle UTF-8 anywhere anything approaching text could be, your program is malformed and should be deprecated and removed. if you wrote code that

if you cannot handle UTF-8 anywhere anything approaching text could be, your program is malformed and should be deprecated and removed. if you wrote code that couldn't handle bob;>/hacked in a username, you would and should be laughed at. why are we using this ancient stuff?

knorker 2y ago
It's not just programs. And it's not just semantics of all-numeric username. It's also whether you want usernames that you cannot type, nor possibly even render. Definitely you can't spell it to someone else. Who owns that file? Oh, it's right-to-left non breaking space smiley snowman Chinese sign for water, I love that guy!
- abigail95 2y ago
  If people want to set up a Debian environment where people are mixing RTL and Hanzi I see no reason for that to be prohibited. Debian has opinions but I disagree that they should extend that far. If my employee Zalgo-fies everything. I don't file a bug report with Debian. I just fire them.
  - Muromec 2y ago
    >If my employee Zalgo-fies everything. I don't file a bug report with Debian. I just fire them. Which such clearly north American attitude you can as well use ASCII for everything.
- Izkata 2y ago
  > Who owns that file? Oh, it's right-to-left non breaking space smiley snowman Chinese sign for water, I love that guy! This reminds me, around 10 years ago on the chat app we used at work, we were able to change our nicknames and I made mine start with a combining character instead of a regular character. No one could ping me, it broke that part of the UI when they tried.
- adrian_b 2y ago
  This thread like also the parent thread is full of comments which are completely outdated, because there already exist standards for Unicode identifiers and obviously they forbid such cases. See e.g. RFC 8264. Only a restricted set of characters is permitted in identifiers, mostly letters and digits. This is enough to write any user name, without allowing "smiley snowman Chinese sign for water" or other such nonsense.
  - knorker 2y ago
    Are you contradicting my extreme example, or all "non-ASCII" characters? I also cannot write something as ordinary as "ç", no more than I can write Chinese character for water. "ç" is a letter. As is "å". The latter is in no way whatsoever related to "a". Certainly there are people with a "ò" in their name, who will tell you that if you spell it "o" then "that's not my name". Your reference to RFC8264 also makes another good point. 38 pages of best practices, plus thousands of pages pulled in from references. Not exactly a case for saying it's feasible for all software to handle this correctly. I did a quick scan of RFC8264, and could not tell if "ç", "å", or "习" are "letters". Yes, that's due to not really trying, since "all I have to do" is read thousands of pages on unicode definitions. What exactly are you trying to say is already established, under the assumption that Unicode already defines identifiers? 7-bit ASCII letters & numbers mostly, or a broader definition of letters? In any case I don't think people are making "outdated" comments, but that they are disagreeing, possibly unicode specs be damned. E.g. see the article itself, where people say that anything that isn't colon or NUL byte should be allowed. > This is enough to write any user name, without allowing "smiley snowman Chinese sign for water" But not to have the username "习近平", right? Or am I misinterpreting by getting lost in references to references in unicode specs?
drtgh 2y ago
With Unicode the same grapheme can be written with a sequence of one or more code points, and each code point can be a sequence of one or more code units. For example "å" can be written with U+00E5, and the same visual glyph "å" with U+0061 + U+030A ( U+0061 {a} plus the code unit U+030A {Combining Ring Above}). Another homoglyph Unicode user name example: · is Café == Café ? · C + a + f + e + '́ ' vs C + a + f + é · Utf8: 43616665CC81 vs 436166C3A9 As one user has pointed out in another comment, some kind of standardisation for that specific use case with some kind of normalisation would be needed first (nevertheless a database search would want a different one, and so on). The above examples are among the simpler ones, there are also unprintable characters, etc. It can be done as in "nothing is impossible", but it's not that easy, it's actually complex.
- abigail95 2y ago
  If a user picks a presentation layer that displays å from noncomparable alphabets, but has them look identical - that's a choice they can and should be able to make. I think it's dumb but I'm not here to hold anyones hand. It's the users choice whether 43616665CC81 == 436166C3A9, same for Café == Café. But they are distinct and separate choices. Text and bytes are separate things. We accept that case sensitivity exists and whether a user/business/program treats them as identical is and should always be their choice to make. There is abstract complexity in the problem, but the context in which text is used solves most of that. If I have handwritten notes and I make a copy but write the second one in cursive and ask someone if they say the same thing - the correct answer isn't "we need to create a standard to normalize the presentation of text" - it's "be more precise in what you are asking". Whether Café == Café depends on if it's written on a road sign, or a network packet with a fixed byte size. Unprintable characters are not text and should not be stored in text fields. Neither are control characters, and as far as I'm concerned should not be included in any text encoding standard. Formatting and terminal processing should never be stored in-band, that's an obvious design flaw that should be corrected. We already deal with ambiguity within ASCII re I vs l vs 1. Some fonts render those identically - Using those fonts in a passport is bad design. Saying we should avoid having to compare those characters at all because some people/systems might confuse them is misguided. This isn't a true rebuttal of what you were saying but some of my next thoughts.
anon-3988 2y ago
Nah, you can use whatever you want for _display_. We have our tower of babel here and we are telling people not to use it? I am not even native English user btw. Having a lingua franca allowed me to understand someone from Russia, China, Japan, etc. Maybe once we have easily accessible ML translate nuances in one language to another without loss we can all talk in our own languages and just translate each others words.
- abigail95 2y ago
  I think people should be able to configure systems to handle a broad range of text from popular encoding standards like UTF-8. Limiting text-space because of communcation is a strange objection that I don't think will hold up over time.
  - numpad0 2y ago
    Unicode is a garbage standard that breaks apart so easily. That's why people hate ideas like yours. You're right in an ideal world but not in this baseline reality with Unicode.
    - adrian_b 2y ago
      Except that it is much better than anything that had existed before it. The earlier handling of non-English alphabets or writing systems was horrible in MS-DOS and Windows. While there have been made some serious mistakes in the development of Unicode, its main principles were fine and it does not have any competition. Feel free to propose and implement a better standard.
      - numpad0 2y ago
        Just skim other branches of this tree. Unicode is non-canonical in many ways. You can't guarantee that the same binary representation reproduce on every machines. That kind of encoding system has no place "under the hood". That should be obvious.
PhilipRoman 2y ago
I really love this powerless use of "should". If you spit on billions of lines of code, all you will get is a dry mouth. The reality defines "what is", unless you have lots of tanks and people under your control, in which case you can change the reality. There is tons of useful code which you will likely never encounter, that helps people accomplish their tasks every day. Do you think there is some central authority who is going to go building to building and dd if=/dev/zero every shell script they find?
- abigail95 2y ago
  This is a contemporary discussion, today, concerning hundreds perhaps thousands of lines of code. That's it. If someone is objecting to changes because of things like "bob;>/hacked". That is laughable, and I will continue to point and laugh. Imagine limiting URL encoding because of SQL injection. We can fix this, then fix the things that break - and then we can improve. Or we can ossify into stone. Your choice.
  - PhilipRoman 2y ago
    >if you cannot handle UTF-8 anywhere anything approaching text could be, your program is malformed and should be deprecated and removed. I was referring to this. Don't get me wrong, I also would love to make sweeping changes to many things in computing. I still think it is perfectly valid to impose reasonable limitations on input even if the program could theoretically handle it - it prevents all kinds of problems at the very root (like allocating disproportionate amounts of resources, infinite timeouts, etc).

knorker 2y ago

It's not just programs. And it's not just semantics of all-numeric username. It's also whether you want usernames that you cannot type, nor possibly even render. Definitely you can't spell it to someone else. Who owns that file? Oh, it's right-to-left non breaking space smiley snowman Chinese sign for water, I love that guy!

abigail95 2y ago
If people want to set up a Debian environment where people are mixing RTL and Hanzi I see no reason for that to be prohibited. Debian has opinions but I disagree that they should extend that far. If my employee Zalgo-fies everything. I don't file a bug report with Debian. I just fire them.
- Muromec 2y ago
  >If my employee Zalgo-fies everything. I don't file a bug report with Debian. I just fire them. Which such clearly north American attitude you can as well use ASCII for everything.
Izkata 2y ago
> Who owns that file? Oh, it's right-to-left non breaking space smiley snowman Chinese sign for water, I love that guy! This reminds me, around 10 years ago on the chat app we used at work, we were able to change our nicknames and I made mine start with a combining character instead of a regular character. No one could ping me, it broke that part of the UI when they tried.
adrian_b 2y ago
This thread like also the parent thread is full of comments which are completely outdated, because there already exist standards for Unicode identifiers and obviously they forbid such cases. See e.g. RFC 8264. Only a restricted set of characters is permitted in identifiers, mostly letters and digits. This is enough to write any user name, without allowing "smiley snowman Chinese sign for water" or other such nonsense.
- knorker 2y ago
  Are you contradicting my extreme example, or all "non-ASCII" characters? I also cannot write something as ordinary as "ç", no more than I can write Chinese character for water. "ç" is a letter. As is "å". The latter is in no way whatsoever related to "a". Certainly there are people with a "ò" in their name, who will tell you that if you spell it "o" then "that's not my name". Your reference to RFC8264 also makes another good point. 38 pages of best practices, plus thousands of pages pulled in from references. Not exactly a case for saying it's feasible for all software to handle this correctly. I did a quick scan of RFC8264, and could not tell if "ç", "å", or "习" are "letters". Yes, that's due to not really trying, since "all I have to do" is read thousands of pages on unicode definitions. What exactly are you trying to say is already established, under the assumption that Unicode already defines identifiers? 7-bit ASCII letters & numbers mostly, or a broader definition of letters? In any case I don't think people are making "outdated" comments, but that they are disagreeing, possibly unicode specs be damned. E.g. see the article itself, where people say that anything that isn't colon or NUL byte should be allowed. > This is enough to write any user name, without allowing "smiley snowman Chinese sign for water" But not to have the username "习近平", right? Or am I misinterpreting by getting lost in references to references in unicode specs?

drtgh 2y ago

With Unicode the same grapheme can be written with a sequence of one or more code points, and each code point can be a sequence of one or more code units. For example "å" can be written with U+00E5, and the same visual glyph "å" with U+0061 + U+030A ( U+0061 {a} plus the code unit U+030A {Combining Ring Above}). Another homoglyph Unicode user name example: · is Café == Café ? · C + a + f + e + '́ ' vs C + a + f + é · Utf8: 43616665CC81 vs 436166C3A9 As one user has pointed out in another comment, some kind of standardisation for that specific use case with some kind of normalisation would be needed first (nevertheless a database search would want a different one, and so on). The above examples are among the simpler ones, there are also unprintable characters, etc. It can be done as in "nothing is impossible", but it's not that easy, it's actually complex.

abigail95 2y ago
If a user picks a presentation layer that displays å from noncomparable alphabets, but has them look identical - that's a choice they can and should be able to make. I think it's dumb but I'm not here to hold anyones hand. It's the users choice whether 43616665CC81 == 436166C3A9, same for Café == Café. But they are distinct and separate choices. Text and bytes are separate things. We accept that case sensitivity exists and whether a user/business/program treats them as identical is and should always be their choice to make. There is abstract complexity in the problem, but the context in which text is used solves most of that. If I have handwritten notes and I make a copy but write the second one in cursive and ask someone if they say the same thing - the correct answer isn't "we need to create a standard to normalize the presentation of text" - it's "be more precise in what you are asking". Whether Café == Café depends on if it's written on a road sign, or a network packet with a fixed byte size. Unprintable characters are not text and should not be stored in text fields. Neither are control characters, and as far as I'm concerned should not be included in any text encoding standard. Formatting and terminal processing should never be stored in-band, that's an obvious design flaw that should be corrected. We already deal with ambiguity within ASCII re I vs l vs 1. Some fonts render those identically - Using those fonts in a passport is bad design. Saying we should avoid having to compare those characters at all because some people/systems might confuse them is misguided. This isn't a true rebuttal of what you were saying but some of my next thoughts.

anon-3988 2y ago

Nah, you can use whatever you want for _display_. We have our tower of babel here and we are telling people not to use it? I am not even native English user btw. Having a lingua franca allowed me to understand someone from Russia, China, Japan, etc. Maybe once we have easily accessible ML translate nuances in one language to another without loss we can all talk in our own languages and just translate each others words.

abigail95 2y ago
I think people should be able to configure systems to handle a broad range of text from popular encoding standards like UTF-8. Limiting text-space because of communcation is a strange objection that I don't think will hold up over time.
- numpad0 2y ago
  Unicode is a garbage standard that breaks apart so easily. That's why people hate ideas like yours. You're right in an ideal world but not in this baseline reality with Unicode.
  - adrian_b 2y ago
    Except that it is much better than anything that had existed before it. The earlier handling of non-English alphabets or writing systems was horrible in MS-DOS and Windows. While there have been made some serious mistakes in the development of Unicode, its main principles were fine and it does not have any competition. Feel free to propose and implement a better standard.
    - numpad0 2y ago
      Just skim other branches of this tree. Unicode is non-canonical in many ways. You can't guarantee that the same binary representation reproduce on every machines. That kind of encoding system has no place "under the hood". That should be obvious.

PhilipRoman 2y ago

I really love this powerless use of "should". If you spit on billions of lines of code, all you will get is a dry mouth. The reality defines "what is", unless you have lots of tanks and people under your control, in which case you can change the reality. There is tons of useful code which you will likely never encounter, that helps people accomplish their tasks every day. Do you think there is some central authority who is going to go building to building and dd if=/dev/zero every shell script they find?

abigail95 2y ago
This is a contemporary discussion, today, concerning hundreds perhaps thousands of lines of code. That's it. If someone is objecting to changes because of things like "bob;>/hacked". That is laughable, and I will continue to point and laugh. Imagine limiting URL encoding because of SQL injection. We can fix this, then fix the things that break - and then we can improve. Or we can ossify into stone. Your choice.
- PhilipRoman 2y ago
  >if you cannot handle UTF-8 anywhere anything approaching text could be, your program is malformed and should be deprecated and removed. I was referring to this. Don't get me wrong, I also would love to make sweeping changes to many things in computing. I still think it is perfectly valid to impose reasonable limitations on input even if the program could theoretically handle it - it prevents all kinds of problems at the very root (like allocating disproportionate amounts of resources, infinite timeouts, etc).