5 ms·

Perl6 extends (NFC normalized) UTF-32 with autogenerated 'synthetic' codpoints for grapheme clusters. This yields an upper bound for the number of unique multi-

by cygx 9y ago

Perl6 extends (NFC normalized) UTF-32 with autogenerated 'synthetic' codpoints for grapheme clusters. This yields an upper bound for the number of unique multi-codepoint grapheme clusters a Perl6 program may encounter (something like 2^31 as I believe only negative numbers are used for synthetics).

ktRolster 9y ago
Since this representation is internal, it means that if there is ever a need for more clusters, they can switch to 64 bit integers without breaking any compatibility.
- cygx 9y ago
  Correct. I should also have clarified that my comment applies to Rakudo Perl6 running on MoarVM. As far as I'm aware, other backends are still behind in implementing proper Unicode semantics.
kibwen 9y ago
Is there a document somewhere describing Perl6's string implementation more precisely? To the best of my knowledge this can't be done, since not only is it legal for combining characters to appear an unlimited number of times in a single grapheme cluster, even discounting repetition the number of legal combinations of single combining characters is surely larger than 2^31. It seems self-evident that if all theoretically-legal combinations of codepoints fit into a 32-bit space, then we just wouldn't bother with combining characters and would use precomposed glyphs exclusively!
- cygx 9y ago
  Not everything that's theoretically unsound is a problem in practice (eg a 128-bit address space would clearly be large enough - there are only so many atoms on the surface of the earth, and the number of glyphs written by humans is quite a bit smaller than that...)
  - kibwen 9y ago
    This is exactly what I'm asking though; if they decided to treat some bespoke combinations of combining characters as though they were precomposed, then I'm curious how they generated that list, and I'm curious what fallback behavior it has for grapheme clusters that aren't specifically accounted for in this way. I'm also sort of curious what their rationale is for doing any of this in the first place, since it seems like the benefit of constant-time indexing isn't especially useful given the memory tradeoffs.
    - cygx 9y ago
      I'm curious how they generated that list Synthetic codepoints get dynamically assigned as they are encountered during program execution. I'm also sort of curious what their rationale is for doing any of this in the first place The rational is constant-time indexing at grapheme cluster level, with the regex engine probably part of the motivation.
    - Someone 9y ago
      From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'. That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
      - cygx 9y ago
        those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs Indeed: It's not an exchange format, but the format of the internal in-memory string representation.
        kibwen 9y ago
        If it's only intended as an in-memory representation, then how about a 24-bit representation, which is just as likely to work "in practice, if not in theory" as the 32-bit representation, while saving 25% of the memory? That gives you 16 million unique grapheme clusters per program instance, which, heck, is probably still more than 99% of programs use; you could easily get away with a 16-bit representation and use no more memory for strings than Java, C#, or Javascript. It's not like using a unique superset of UTF-32 is saving anyone the hassle of needing to convert between formats, since the vast majority of the world is ASCII-ish, UTF-8, and UTF-16 anyway.
- b2gills 9y ago
  You're right it can't be done (Perl 6 does that a lot). The feature could be made to start a new mapping of synthetic characters before it runs out, and throw away the mapping when the last of the old strands gets GCed. Basically it is a problem in theory, but might not be in practice.

by cygx 9y ago

ktRolster 9y ago

Since this representation is internal, it means that if there is ever a need for more clusters, they can switch to 64 bit integers without breaking any compatibility.

cygx 9y ago
Correct. I should also have clarified that my comment applies to Rakudo Perl6 running on MoarVM. As far as I'm aware, other backends are still behind in implementing proper Unicode semantics.

kibwen 9y ago

Is there a document somewhere describing Perl6's string implementation more precisely? To the best of my knowledge this can't be done, since not only is it legal for combining characters to appear an unlimited number of times in a single grapheme cluster, even discounting repetition the number of legal combinations of single combining characters is surely larger than 2^31. It seems self-evident that if all theoretically-legal combinations of codepoints fit into a 32-bit space, then we just wouldn't bother with combining characters and would use precomposed glyphs exclusively!

cygx 9y ago
Not everything that's theoretically unsound is a problem in practice (eg a 128-bit address space would clearly be large enough - there are only so many atoms on the surface of the earth, and the number of glyphs written by humans is quite a bit smaller than that...)
- kibwen 9y ago
  This is exactly what I'm asking though; if they decided to treat some bespoke combinations of combining characters as though they were precomposed, then I'm curious how they generated that list, and I'm curious what fallback behavior it has for grapheme clusters that aren't specifically accounted for in this way. I'm also sort of curious what their rationale is for doing any of this in the first place, since it seems like the benefit of constant-time indexing isn't especially useful given the memory tradeoffs.
  - cygx 9y ago
    I'm curious how they generated that list Synthetic codepoints get dynamically assigned as they are encountered during program execution. I'm also sort of curious what their rationale is for doing any of this in the first place The rational is constant-time indexing at grapheme cluster level, with the regex engine probably part of the motivation.
  - Someone 9y ago
    From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'. That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
    - cygx 9y ago
      those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs Indeed: It's not an exchange format, but the format of the internal in-memory string representation.
      - kibwen 9y ago
        If it's only intended as an in-memory representation, then how about a 24-bit representation, which is just as likely to work "in practice, if not in theory" as the 32-bit representation, while saving 25% of the memory? That gives you 16 million unique grapheme clusters per program instance, which, heck, is probably still more than 99% of programs use; you could easily get away with a 16-bit representation and use no more memory for strings than Java, C#, or Javascript. It's not like using a unique superset of UTF-32 is saving anyone the hassle of needing to convert between formats, since the vast majority of the world is ASCII-ish, UTF-8, and UTF-16 anyway.
b2gills 9y ago
You're right it can't be done (Perl 6 does that a lot). The feature could be made to start a new mapping of synthetic characters before it runs out, and throw away the mapping when the last of the old strands gets GCed. Basically it is a problem in theory, but might not be in practice.