4 ms·
This is an interesting article overall, but I'm unsure what this part is trying to say: > Perl 6 is aware of graphemes and combining codepoints, and unlike Pyt
by kibwen 9y ago
This is an interesting article overall, but I'm unsure what this part is trying to say:
> Perl 6 is aware of graphemes and combining codepoints, and unlike Python, Swift, or Elixir, Perl 6 can can access arbitrary graphemes by position in constant time, rather than iterating over every grapheme to access a particular one. [...] Graphemes are stored internally as 32-bit integers; this address space is large enough to contain all legal codepoint combinations
It sounds like Perl6 is simply storing strings as UTF-32. AFAIK, there's nothing called a "grapheme" in Unicode; the closest I know of are "grapheme clusters", which are theoretically unbounded in size and so cannot simply be stored in a 32-bit integer. Maybe by "grapheme" the author means "Unicode scalar value"? But being able to access those in constant time isn't especially useful, AFAIK.
- deleted 9y ago[deleted]
- cygx 9y agoPerl6 extends (NFC normalized) UTF-32 with autogenerated 'synthetic' codpoints for grapheme clusters. This yields an upper bound for the number of unique multi-codepoint grapheme clusters a Perl6 program may encounter (something like 2^31 as I believe only negative numbers are used for synthetics).
- ktRolster 9y agoSince this representation is internal, it means that if there is ever a need for more clusters, they can switch to 64 bit integers without breaking any compatibility.
- cygx 9y agoCorrect. I should also have clarified that my comment applies to Rakudo Perl6 running on MoarVM. As far as I'm aware, other backends are still behind in implementing proper Unicode semantics.
- kibwen 9y agoIs there a document somewhere describing Perl6's string implementation more precisely? To the best of my knowledge this can't be done, since not only is it legal for combining characters to appear an unlimited number of times in a single grapheme cluster, even discounting repetition the number of legal combinations of single combining characters is surely larger than 2^31. It seems self-evident that if all theoretically-legal combinations of codepoints fit into a 32-bit space, then we just wouldn't bother with combining characters and would use precomposed glyphs exclusively!
- cygx 9y agoNot everything that's theoretically unsound is a problem in practice (eg a 128-bit address space would clearly be large enough - there are only so many atoms on the surface of the earth, and the number of glyphs written by humans is quite a bit smaller than that...)
- kibwen 9y agoThis is exactly what I'm asking though; if they decided to treat some bespoke combinations of combining characters as though they were precomposed, then I'm curious how they generated that list, and I'm curious what fallback behavior it has for grapheme clusters that aren't specifically accounted for in this way. I'm also sort of curious what their rationale is for doing any of this in the first place, since it seems like the benefit of constant-time indexing isn't especially useful given the memory tradeoffs.
- cygx 9y agoI'm curious how they generated that list Synthetic codepoints get dynamically assigned as they are encountered during program execution. I'm also sort of curious what their rationale is for doing any of this in the first place The rational is constant-time indexing at grapheme cluster level, with the regex engine probably part of the motivation.
- Someone 9y agoFrom what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'. That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
- samcv 9y agoMoarVM stores strings as either 32-bit signed or 8-bit signed. Negative numbers are reserved for "synthetic's" which are for grapheme clusters which contain more than a single codepoint. Synthetics are allocated as needed, so we can store as much as we need to. The only issue would be storing more than 2^31 possible combinations… though you would run out of memory before you ran out of negative numbers in the 32-bit integer space.
- kibwen 9y agoCool, thanks for the info. Should I assume from the bevy of comments rather than links that this isn't documented anywhere? If both myself and the OP had to ask the devs personally for this info, then it may be productive to make a blog post describing your strategy (or just a comment in the source, even).
- samcv 9y agoYou might want to see some of my slides here: https://cry.nu/YAPC-EU-2017/MoarVM-Internals/#/9 https://cry.nu/YAPC-EU-2017/MoarVM-Internals/#/9 Link is where the string implementation starts. I don't see it detailed in our documentation, but it may be somewhere. As part of my Unicode grant from the Perl Foundation I will try and add some implementation information to the MoarVM docs. Let me know if you have anymore questions, Thanks!
- raiph 9y agoDoes https://github.com/MoarVM/MoarVM/search?utf8=%E2%9C%93&q=grapheme&type= https://github.com/MoarVM/MoarVM/search?utf8=%E2%9C%93&q=gra... help?
- raiph 9y agoPer the Unicode standard, "grapheme" means "what a user thinks of as a character", i.e. nothing to do with computers, whereas "grapheme clusters" are sequences of codepoints that "approximate" a grapheme (more approximately if it's EGC, less approximately if it's TGC).[1] Being able to access characters or sub-strings in constant time is useful for some (all?) character and sub-string processing. [1] http://www.unicode.org/glossary/#grapheme http://www.unicode.org/glossary/#grapheme and http://www.unicode.org/glossary/#grapheme_cluster http://www.unicode.org/glossary/#grapheme_cluster