3 ms·

Oil has better Unicode support than Python 2 or 3 for shell-like tasks, because of the way file systems, libc, and the kernel work. Explained here: http://www

by chubot 7y ago

Oil has better Unicode support than Python 2 or 3 for shell-like tasks, because of the way file systems, libc, and the kernel work. Explained here: http://www.oilshell.org/blog/2018/03/04.html#faq http://www.oilshell.org/blog/2018/03/04.html#faq which links: http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ (by Armin Ronacher) The summary is that there are two kinds of Unicode support: - UTF-8 based: Go, Rust, and Oil (and Perl 5 it seems, not sure about Raku) - array-of-codepoint based: bash/zsh/libc, Python, Java, JavaScript, Windows (JavaScript notably requires surrogate pairs, Python used to have build time configuration, now has complex storage heuristics, etc.) This isn't a theoretical problem -- the Unicode problems in the other comment thread I linked are real and show up in practice.

lizmat 7y ago
Unicode support of Raku is grapheme based (NFG or Normalization Form Grapheme). The unicode theory: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... , about the implementation: https://6guts.wordpress.com/2015/04/12/this-week-unicode-normalization-many-rts/ https://6guts.wordpress.com/2015/04/12/this-week-unicode-nor...
Grinnz 7y ago
Perl (5) is array-of-codepoint based, at the logical level. Those codepoints might be internally stored as their encoded-to-UTF-8-bytes, or they might not, but this does not affect the usage of the string. Many don't really understand the string model (because many don't really understand character encoding) but it comes down to: all input and output is going to be bytes, which by default is stored as the codepoints sharing the ordinals of those bytes, and there are several mechanisms by which you can manually or automatically decode/encode those byte ordinals to the represented characters; for most text processing, you do this on STDIN and STDOUT.

by chubot 7y ago

lizmat 7y ago

Unicode support of Raku is grapheme based (NFG or Normalization Form Grapheme). The unicode theory: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... , about the implementation: https://6guts.wordpress.com/2015/04/12/this-week-unicode-normalization-many-rts/ https://6guts.wordpress.com/2015/04/12/this-week-unicode-nor...

Grinnz 7y ago

Perl (5) is array-of-codepoint based, at the logical level. Those codepoints might be internally stored as their encoded-to-UTF-8-bytes, or they might not, but this does not affect the usage of the string. Many don't really understand the string model (because many don't really understand character encoding) but it comes down to: all input and output is going to be bytes, which by default is stored as the codepoints sharing the ordinals of those bytes, and there are several mechanisms by which you can manually or automatically decode/encode those byte ordinals to the represented characters; for most text processing, you do this on STDIN and STDOUT.