3 ms·

If you use a 32-bit scheme, you can dynamically assign multi-character (extended) grapheme clusters to unused code units to get a fixed-width encoding. Perl6 c

by cygx 11y ago

If you use a 32-bit scheme, you can dynamically assign multi-character (extended) grapheme clusters to unused code units to get a fixed-width encoding. Perl6 calls this NFG [1]. [1] http://design.perl6.org/S15.html http://design.perl6.org/S15.html ^ link currently broken, the plain-text version is at https://raw.githubusercontent.com/perl6/specs/master/S15-unicode.pod https://raw.githubusercontent.com/perl6/specs/master/S15-uni...

lmm 11y ago
You can't use that for storage. > The mapping between negative numbers and graphemes in this form is not guaranteed constant, even between strings in the same process.
- cygx 11y ago
  What's your storage requirement that's not adequately solved by the existing encoding schemes?
  - lmm 11y ago
    What are you suggesting, store strings in UTF8 and then "normalize" them into this bizarre format whenever you load/save them purely so that offsets correspond to grapheme clusters? Doesn't seem worth the overhead to my eyes.
    - cygx 11y ago
      In-memory string representation rarely corresponds to on-disk representation. Various programming languages (Java, C#, Objective-C, JavaScript, ...) as well as some well-known libraries (ICU, Windows API, Qt) use UTF-16 internally. How much data do you have lying around that's UTF-16? Sure, more recently, Go and Rust have decided to go with UTF-8, but that's far from common, and it does have some drawbacks compared to the Perl6 (NFG) or Python3 (latin-1, UCS-2, UCS-4 as appropriate) model if you have to do actual processing instead of just passing opaque strings around. Also note that you have to go through a normalization step anyway if you don't want to be tripped up by having multiple ways to represent a single grapheme.
    - raiph 11y ago
      NFG enables O(N) algorithms for character level operations. The overhead is entirely wasted on code that does no character level operations. For code that does do some character level operations, avoiding quadratic behavior may pay off handsomely.

by cygx 11y ago

lmm 11y ago

You can't use that for storage. > The mapping between negative numbers and graphemes in this form is not guaranteed constant, even between strings in the same process.

cygx 11y ago
What's your storage requirement that's not adequately solved by the existing encoding schemes?
- lmm 11y ago
  What are you suggesting, store strings in UTF8 and then "normalize" them into this bizarre format whenever you load/save them purely so that offsets correspond to grapheme clusters? Doesn't seem worth the overhead to my eyes.
  - cygx 11y ago
    In-memory string representation rarely corresponds to on-disk representation. Various programming languages (Java, C#, Objective-C, JavaScript, ...) as well as some well-known libraries (ICU, Windows API, Qt) use UTF-16 internally. How much data do you have lying around that's UTF-16? Sure, more recently, Go and Rust have decided to go with UTF-8, but that's far from common, and it does have some drawbacks compared to the Perl6 (NFG) or Python3 (latin-1, UCS-2, UCS-4 as appropriate) model if you have to do actual processing instead of just passing opaque strings around. Also note that you have to go through a normalization step anyway if you don't want to be tripped up by having multiple ways to represent a single grapheme.
  - raiph 11y ago
    NFG enables O(N) algorithms for character level operations. The overhead is entirely wasted on code that does no character level operations. For code that does do some character level operations, avoiding quadratic behavior may pay off handsomely.