5 ms·

Perhaps it's time to agree upon how to Unicode in identifiers? The normalization, unprintable characters, confusing characters with same glyphs, etc. It's obvio

by rini17 2y ago

Perhaps it's time to agree upon how to Unicode in identifiers? The normalization, unprintable characters, confusing characters with same glyphs, etc. It's obviously problematic when everyone is doing it on their own.

magicalhippo 2y ago
As long as I can enter my Zalgo[1] username, I'm fine with your suggestion. [1]: https://en.wikipedia.org/wiki/Zalgo_text https://en.wikipedia.org/wiki/Zalgo_text
m000 2y ago
Good luck bringing everyone together. There's still a ton of Microsoft software that relies on the presence of the BOM [1], despite practically everyone else not using it. And bidirectional rsync between practically everything else and a Mac still requires `--iconv=utf-8,utf-8-mac` to avoid problems because of homographs. [1] https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark
bayindirh 2y ago
The first bar to clear is "The Turkish Test"[0], then we can talk about Unicode. It'll smooth the rest of the process a lot. You can't guess how many workarounds I implement to make sure that a stray application doesn't get "ı" or "İ" in their naive codepaths, and start burning mid-flight (e.g.: Kodi, Pagico, some old Java programs, oh my...). [0]: https://blog.codinghorror.com/whats-wrong-with-turkey/ https://blog.codinghorror.com/whats-wrong-with-turkey/
- beardyw 2y ago
  The date format part is ridiculous. Americans are almost unique in using mm/dd/yyyy, so an assumption of that would be plain wrong.
  - bayindirh 2y ago
    Localization libraries handle these parts well, since date is same with Europe (and generally stored as time-date objects rather than pure strings). None of the number shenanigans cause problems since these numbers are always stored as IEEE754 or other decimal formats. Money is no problem as well. However, when you go through an upper() or lower() or anything which plays with capitalization, and if that data is being fed to a hash algorithm or anything which mucks with strings, boy, oh boy... The easiest way is to sanitize these programmatic parts with forced locale of en_US or plain old "C". If the strings is not facing to the user and never localized, just force its locale. It's the only sane way.
    - oblio 2y ago
      > since date is same with Europe Do you mean MM-DD-YYYY? No, the vast majority of Europe does DD-MM-YYYY in some form or another.
      - bayindirh 2y ago
        No, I mean DD-MM-YYYY. We use the same format with the vast majority of Europe.
        oblio 2y ago
        I'm confused, are you talking about the US? The US for sure does not default to DD-MM-YYYY.
        bayindirh 2y ago
        I'm talking about the Turkish Language, its locale and its peculiarities since it has letters "ı" and "İ". So, I'm talking about Turkish date format. Turkey uses DD-MM-YYYY format, like the most of the Europe.
  - a3w 2y ago
    https://xkcd.com/1179/ https://xkcd.com/1179/ I heard the US and A are moving to the hissing cat date format shown here.
throw0101a 2y ago
> Perhaps it's time to agree upon how to Unicode in identifiers? And then update all data structures that refer to them (like last and w/who, also NFS), as well as file formats (like cpio, tar, and pax which encodes ownership).
- maccard 2y ago
  Yes. Those formats have had 20 years since Unicode was standardised, and things like my terminal still routinely break when given “unexpected” inputs. Practically every other application can handle it.
layer8 2y ago
Unicode has provided a specification for Unicode identifiers since 2005: https://www.unicode.org/reports/tr31/ https://www.unicode.org/reports/tr31/
- rini17 2y ago
  Great! Is there a library for their validation? ICU seems to have only spoof checker for confusables.
  - rurban 2y ago
    libu8ident
    - westurner 2y ago
      rurban/libu8ident : https://github.com/rurban/libu8ident https://github.com/rurban/libu8ident : > unicode security guidelines for identifiers
  - westurner 2y ago
    ICU: International Components for Unicode: https://en.wikipedia.org/wiki/International_Components_for_Unicode https://en.wikipedia.org/wiki/International_Components_for_U... unicode-org/icu: https://github.com/unicode-org/icu https://github.com/unicode-org/icu Microsoft/ICU: https://github.com/microsoft/icu https://github.com/microsoft/icu IDN: Internationalized domain name: https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Internationalized_domain_name Punycode: https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode IDN homograph attack: https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack CWE-1007: Insufficient Visual Distinction of Homoglyphs Presented to User: https://cwe.mitre.org/data/definitions/1007.html https://cwe.mitre.org/data/definitions/1007.html GNU libidn/libidn2: https://gitlab.com/libidn/libidn2 https://gitlab.com/libidn/libidn2 Comparison of regular expression engines > Language features > Part 2; Unicode support: https://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines#Part_2 https://en.wikipedia.org/wiki/Comparison_of_regular_expressi...
secondcoming 2y ago
Would punycode be suitable?

by rini17 2y ago

magicalhippo 2y ago

As long as I can enter my Zalgo[1] username, I'm fine with your suggestion. [1]: https://en.wikipedia.org/wiki/Zalgo_text https://en.wikipedia.org/wiki/Zalgo_text

m000 2y ago

Good luck bringing everyone together. There's still a ton of Microsoft software that relies on the presence of the BOM [1], despite practically everyone else not using it. And bidirectional rsync between practically everything else and a Mac still requires `--iconv=utf-8,utf-8-mac` to avoid problems because of homographs. [1] https://en.wikipedia.org/wiki/Byte_order_mark https://en.wikipedia.org/wiki/Byte_order_mark

bayindirh 2y ago

The first bar to clear is "The Turkish Test"[0], then we can talk about Unicode. It'll smooth the rest of the process a lot. You can't guess how many workarounds I implement to make sure that a stray application doesn't get "ı" or "İ" in their naive codepaths, and start burning mid-flight (e.g.: Kodi, Pagico, some old Java programs, oh my...). [0]: https://blog.codinghorror.com/whats-wrong-with-turkey/ https://blog.codinghorror.com/whats-wrong-with-turkey/

beardyw 2y ago
The date format part is ridiculous. Americans are almost unique in using mm/dd/yyyy, so an assumption of that would be plain wrong.
- bayindirh 2y ago
  Localization libraries handle these parts well, since date is same with Europe (and generally stored as time-date objects rather than pure strings). None of the number shenanigans cause problems since these numbers are always stored as IEEE754 or other decimal formats. Money is no problem as well. However, when you go through an upper() or lower() or anything which plays with capitalization, and if that data is being fed to a hash algorithm or anything which mucks with strings, boy, oh boy... The easiest way is to sanitize these programmatic parts with forced locale of en_US or plain old "C". If the strings is not facing to the user and never localized, just force its locale. It's the only sane way.
  - oblio 2y ago
    > since date is same with Europe Do you mean MM-DD-YYYY? No, the vast majority of Europe does DD-MM-YYYY in some form or another.
    - bayindirh 2y ago
      No, I mean DD-MM-YYYY. We use the same format with the vast majority of Europe.
      - oblio 2y ago
        I'm confused, are you talking about the US? The US for sure does not default to DD-MM-YYYY.
        bayindirh 2y ago
        I'm talking about the Turkish Language, its locale and its peculiarities since it has letters "ı" and "İ". So, I'm talking about Turkish date format. Turkey uses DD-MM-YYYY format, like the most of the Europe.
- a3w 2y ago
  https://xkcd.com/1179/ https://xkcd.com/1179/ I heard the US and A are moving to the hissing cat date format shown here.

throw0101a 2y ago

> Perhaps it's time to agree upon how to Unicode in identifiers? And then update all data structures that refer to them (like last and w/who, also NFS), as well as file formats (like cpio, tar, and pax which encodes ownership).

maccard 2y ago
Yes. Those formats have had 20 years since Unicode was standardised, and things like my terminal still routinely break when given “unexpected” inputs. Practically every other application can handle it.

layer8 2y ago

Unicode has provided a specification for Unicode identifiers since 2005: https://www.unicode.org/reports/tr31/ https://www.unicode.org/reports/tr31/

rini17 2y ago
Great! Is there a library for their validation? ICU seems to have only spoof checker for confusables.
- rurban 2y ago
  libu8ident
  - westurner 2y ago
    rurban/libu8ident : https://github.com/rurban/libu8ident https://github.com/rurban/libu8ident : > unicode security guidelines for identifiers
- westurner 2y ago
  ICU: International Components for Unicode: https://en.wikipedia.org/wiki/International_Components_for_Unicode https://en.wikipedia.org/wiki/International_Components_for_U... unicode-org/icu: https://github.com/unicode-org/icu https://github.com/unicode-org/icu Microsoft/ICU: https://github.com/microsoft/icu https://github.com/microsoft/icu IDN: Internationalized domain name: https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Internationalized_domain_name Punycode: https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode IDN homograph attack: https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack CWE-1007: Insufficient Visual Distinction of Homoglyphs Presented to User: https://cwe.mitre.org/data/definitions/1007.html https://cwe.mitre.org/data/definitions/1007.html GNU libidn/libidn2: https://gitlab.com/libidn/libidn2 https://gitlab.com/libidn/libidn2 Comparison of regular expression engines > Language features > Part 2; Unicode support: https://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines#Part_2 https://en.wikipedia.org/wiki/Comparison_of_regular_expressi...

secondcoming 2y ago

Would punycode be suitable?