4 ms·

Not everything that's theoretically unsound is a problem in practice (eg a 128-bit address space would clearly be large enough - there are only so many atoms on

by cygx 9y ago

Not everything that's theoretically unsound is a problem in practice (eg a 128-bit address space would clearly be large enough - there are only so many atoms on the surface of the earth, and the number of glyphs written by humans is quite a bit smaller than that...)

kibwen 9y ago
This is exactly what I'm asking though; if they decided to treat some bespoke combinations of combining characters as though they were precomposed, then I'm curious how they generated that list, and I'm curious what fallback behavior it has for grapheme clusters that aren't specifically accounted for in this way. I'm also sort of curious what their rationale is for doing any of this in the first place, since it seems like the benefit of constant-time indexing isn't especially useful given the memory tradeoffs.
- cygx 9y ago
  I'm curious how they generated that list Synthetic codepoints get dynamically assigned as they are encountered during program execution. I'm also sort of curious what their rationale is for doing any of this in the first place The rational is constant-time indexing at grapheme cluster level, with the regex engine probably part of the motivation.
- Someone 9y ago
  From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'. That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
  - cygx 9y ago
    those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs Indeed: It's not an exchange format, but the format of the internal in-memory string representation.
    - kibwen 9y ago
      If it's only intended as an in-memory representation, then how about a 24-bit representation, which is just as likely to work "in practice, if not in theory" as the 32-bit representation, while saving 25% of the memory? That gives you 16 million unique grapheme clusters per program instance, which, heck, is probably still more than 99% of programs use; you could easily get away with a 16-bit representation and use no more memory for strings than Java, C#, or Javascript. It's not like using a unique superset of UTF-32 is saving anyone the hassle of needing to convert between formats, since the vast majority of the world is ASCII-ish, UTF-8, and UTF-16 anyway.
      - cygx 9y ago
        > how about a 24-bit representation Because 24-bit integers are not a thing on x86?
        kibwen 9y ago
        Who says you need to use integers? You're completely ignoring my question. Again, if you know of a design doc buried on a mailing list somewhere please just link me to that so that I can answer all of my questions at once.

by cygx 9y ago

kibwen 9y ago

This is exactly what I'm asking though; if they decided to treat some bespoke combinations of combining characters as though they were precomposed, then I'm curious how they generated that list, and I'm curious what fallback behavior it has for grapheme clusters that aren't specifically accounted for in this way. I'm also sort of curious what their rationale is for doing any of this in the first place, since it seems like the benefit of constant-time indexing isn't especially useful given the memory tradeoffs.

cygx 9y ago
I'm curious how they generated that list Synthetic codepoints get dynamically assigned as they are encountered during program execution. I'm also sort of curious what their rationale is for doing any of this in the first place The rational is constant-time indexing at grapheme cluster level, with the regex engine probably part of the motivation.
Someone 9y ago
From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'. That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
- cygx 9y ago
  those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs Indeed: It's not an exchange format, but the format of the internal in-memory string representation.
  - kibwen 9y ago
    If it's only intended as an in-memory representation, then how about a 24-bit representation, which is just as likely to work "in practice, if not in theory" as the 32-bit representation, while saving 25% of the memory? That gives you 16 million unique grapheme clusters per program instance, which, heck, is probably still more than 99% of programs use; you could easily get away with a 16-bit representation and use no more memory for strings than Java, C#, or Javascript. It's not like using a unique superset of UTF-32 is saving anyone the hassle of needing to convert between formats, since the vast majority of the world is ASCII-ish, UTF-8, and UTF-16 anyway.
    - cygx 9y ago
      > how about a 24-bit representation Because 24-bit integers are not a thing on x86?
      - kibwen 9y ago
        Who says you need to use integers? You're completely ignoring my question. Again, if you know of a design doc buried on a mailing list somewhere please just link me to that so that I can answer all of my questions at once.