3 ms·
In Perl 6: $width-in-terminal = $text.chars; $codepoints = $text.codes; $bytes = $text.encode('utf8').bytes; The .chars method should be the faste
by labster 10y ago
In Perl 6:
$width-in-terminal = $text.chars;
$codepoints = $text.codes;
$bytes = $text.encode('utf8').bytes;
The .chars method should be the fastest, because Perl 6 internally uses strings of fully composed characters (normalized form grapheme). It's much better than having to do regex hacks like in Python.
- dom0 10y ago$width-in-terminal = $text.chars; ^- very likely wrong. wcswidth computes the cell-width (or column-count) of a Unicode string, which is unrelated to the count of graphemes, EGCs or code points. For example, Latin characters are one cell/column wide, while for example many CJK characters occupy two cells/columns, while they are still one EGC. A typical application is printing CJK things to a terminal mask, progress display or similar.
- labster 10y agoAh right. I guess we were meaning different things by width. And you're probably more correct here.
- Someone 10y agoGiven that one can make graphemes that contain many Unicode code points (213 in this extreme example: https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_longest_unicode_character/ https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_lon...), I wondered whether that can be correct for any reasonable definition of "fully composed character". As far as I can tell, it turned out to be correct, though. Perl 6 has its own Inocode normalization variant called NFG (https://design.perl6.org/S15.html#NFG https://design.perl6.org/S15.html#NFG): "Formally Perl 6 graphemes are defined exactly according to Unicode Grapheme Cluster Boundaries at level "extended" (in contrast to "tailored" or "legacy"), see Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION 3 Grapheme Cluster Boundaries>. This is the same as the Perl 5 character class \X Match Unicode "eXtended grapheme cluster" With NFG, strings start by being run through the normal NFC process, compressing any given character sequences into precomposed characters. Any graphemes remaining without precomposed characters, such as ậ or नि, are given their own internal designation to refer to them, at least 32 bits in length, in such a way that they avoid clashing with any potential future changes to Unicode. The mapping between these internal designations and graphemes in this form is not guaranteed constant, even between strings in the same process." From that, I guess Perl 6 extends its mapping between NFG code points (an extension of Unicode code points) and Unicode graphemes clusters whenever it encounters a grapheme cluster it hasn't seen before. Ignoring performance concerns (might not be bad, but I'm not sure about that), that seems a nice approach.