4 ms·

> I consider the fact that a stupid-simple package was depended-upon by so many mature libraries as an indication it should be a language feature. Unfortunatel

by mbell 9y ago

> I consider the fact that a stupid-simple package was depended-upon by so many mature libraries as an indication it should be a language feature. Unfortunately neither the npm module nor the browser version really do what most people want and string handling in javascript is still a minefield. '\u{1F4A9}'.padStart(5, '1') => "111" // oops (\u{1F4A9} is at the end of this, HN filter) '\u{1F4A9}'.length => 2 [...'\u{1F4A9}'].length => 1 //WTF? 'mañana'.padStart(7, '1') => "1mañana" // ok 'man\u0303ana'.padStart(7, '1') => "mañana" // oops 'man\u0303ana'.length => 7 [...'man\u0303ana'].length => 7 // WTF? Why doesn't this match the behavior of [...'\u{1F4A9}'].length ? 'man\u0303ana'.normalize('NFC').padStart(7, '1') => "1mañana" // OK I do understand the unicode issues here, but the inconsistency in the APIs from a user perspective and lack of any fully cross browser support for sane string processing in Javascript means we still have only a few options: 1) Don't do string processing in javascript at all. 2) Include a library to make it sane, these are usually huge as they usually need large lookup tables. 3) Accept that things won't always be correct. This is one example but the lack of a sane standard library in Javascript is one of the biggest problems the web has right now. I'd be curious to know how many bytes of JS are loaded on the average website just to work around the lack of standard library support for basic functionality, I'd bet it's a very large number. Another fun one: Try to parse a URL and append extra query params to it, correctly.

javajosh 9y ago
It seems like certain code points (like '\u{1F4A9}' aka the poop emoji) are a single character but the string report a length of 2. That is the root of all of those problems. One of your "problems", the length of an array with a single string element, isn't a problem.
- recursive 9y ago
  > One of your "problems", the length of an array with a single string element That's not what it is. Look again, and pay attention to the ... part.
  - javajosh 9y ago
    That's really helpful, thanks.
- mbell 9y ago
  > One of your "problems", the length of an array with a single string element, isn't a problem. You're misunderstand the code, it's using the spread operator on a string: [...'word'] => ["w", "o", "r", "d"] What is being demonstrated is that under the hood, javascript stores astral plane codepoints as surrogate pairs and strings operate on 'characters' which is why '\u{1F4A9}'.length => 2. But, when the spread operator is applied to a string, it breaks up the string into codepoints, not characters. This is also why [...'man\u0303ana'].length => 7, the combining tilde is a separate codepoint. This is an example of how wonky string processing in JS is, [...string].length is actually the most straightforward way to get a count of codepoints in a string.
- colejohnson66 9y ago
  Any Unicode codepoint above U+FFFF will require 2 UTF-16 characters
jcranmer 9y ago
> [...'man\u0303ana'].length => 7 // WTF? Why doesn't this match the behavior of [...'\u{1F4A9}'].length ? The key thing to remember is that iteration over Unicode strings only makes sense as iteration over code points, not UCS-2 characters, not bytes, not grapheme clusters. The JS String iterator was very deliberately made to iterate over code points. That length reports UCS-2 characters is a historical mistake. That padding is operating on UCS-2 characters is probably a reflection of the fact that the operation isn't well-defined beyond ASCII.
- mbell 9y ago
  > The key thing to remember is that iteration over Unicode strings only makes sense as iteration over code points, not UCS-2 characters, not bytes, not grapheme clusters. There are tons of situations where interating over grapheme clusters is what you want to do.
  - wruza 9y ago
    And tons of situations where you don't want neither of two (e.g. nfd vs. nfc). Cairo graphics library has utilities for text rendering, explicitly called "toy text" functions in reference, leaving serious rendering to Pango. That's fair. Languages should not call unicode strings "unicode strings" if these are not covered in detail by special libraries with distinct names for ucp/ucs/etc lengths, iterators, etc. There is no such thing as string length or "char" anymore. String is blank or non-blank, anything beyond that is too complex to be part of any stdlib. Even "blank" is not so obvious today.
ricardobeat 9y ago
'mañana'.padStart(7, '1') => "1mañana" // ok 'man\u0303ana'.padStart(7, '1') => "mañana" // oops You are being disingenuous here. Those are different strings, with different lengths (try copying this into the console): 'mañana'.length // 6 'mañana'.length // 7 The latter has two stacked characters. These issues are inherent to Unicode and `padStart` is treating the strings correctly. If you need normalization, use the .normalize method you mentioned yourself. This is a major improvement: double-wide and stacked characters have been there since ES3, but now the language is providing standard tools to work with them.
- yarg 9y ago
  I don't think that it's really fair to say that he's being disingenuous here, regardless of the underlying byte form the strings look indistinguisable and often users (devs) will expect them to function as such. I think it would be less confusing to define .length as the number of characters and have an additional .size method returning the number of bytes (I'm assuming that's what .length returns, if not it's even more confusing). Of course, that already wasn't done - meh.
  - mbell 9y ago
    > have an additional .size method returning the number of bytes (I'm assuming that's what .length returns, if not it's even more confusing). It's actually not the number of bytes, it's the number of...'codepoint pieces' is what it could be called I guess? Javascript's language level string implementation is something like UCS-2 with the addition of surrogate pairs being allowed, but counted as separate 'characters' for things like length and index access. It's some twisted middle ground between UCS-2 and UTF-16.
    - yarg 9y ago
      That seems deranged to me. Like a true length calculation, It still requires a complex (albeit cachable) calculation to resolve, but it fails to return the length of the string in terms of the number of characters as they would be natually presented. I understand a need in some contexts to distinguish between a character and its subsequent modifiers - but I do not see such a context here. Design by committee?
tracker1 9y ago
That's why I hated the "it might break stuff" arguments against making "string" interfaces against characters (including combinators), and always using UTF-8 for encoding internally, in memory. Would have made a lot of that easier. As to your last bit, I tend to favor encodeURIComponent and have done it correctly... the main reason, is to avoid "+" vs " " in query strings.
lngnmn 9y ago
Holy fuck! The Fractal Of Bad Design all over again...

by mbell 9y ago

javajosh 9y ago

It seems like certain code points (like '\u{1F4A9}' aka the poop emoji) are a single character but the string report a length of 2. That is the root of all of those problems. One of your "problems", the length of an array with a single string element, isn't a problem.

recursive 9y ago
> One of your "problems", the length of an array with a single string element That's not what it is. Look again, and pay attention to the ... part.
- javajosh 9y ago
  That's really helpful, thanks.
mbell 9y ago
> One of your "problems", the length of an array with a single string element, isn't a problem. You're misunderstand the code, it's using the spread operator on a string: [...'word'] => ["w", "o", "r", "d"] What is being demonstrated is that under the hood, javascript stores astral plane codepoints as surrogate pairs and strings operate on 'characters' which is why '\u{1F4A9}'.length => 2. But, when the spread operator is applied to a string, it breaks up the string into codepoints, not characters. This is also why [...'man\u0303ana'].length => 7, the combining tilde is a separate codepoint. This is an example of how wonky string processing in JS is, [...string].length is actually the most straightforward way to get a count of codepoints in a string.
colejohnson66 9y ago
Any Unicode codepoint above U+FFFF will require 2 UTF-16 characters

jcranmer 9y ago

> [...'man\u0303ana'].length => 7 // WTF? Why doesn't this match the behavior of [...'\u{1F4A9}'].length ? The key thing to remember is that iteration over Unicode strings only makes sense as iteration over code points, not UCS-2 characters, not bytes, not grapheme clusters. The JS String iterator was very deliberately made to iterate over code points. That length reports UCS-2 characters is a historical mistake. That padding is operating on UCS-2 characters is probably a reflection of the fact that the operation isn't well-defined beyond ASCII.

mbell 9y ago
> The key thing to remember is that iteration over Unicode strings only makes sense as iteration over code points, not UCS-2 characters, not bytes, not grapheme clusters. There are tons of situations where interating over grapheme clusters is what you want to do.
- wruza 9y ago
  And tons of situations where you don't want neither of two (e.g. nfd vs. nfc). Cairo graphics library has utilities for text rendering, explicitly called "toy text" functions in reference, leaving serious rendering to Pango. That's fair. Languages should not call unicode strings "unicode strings" if these are not covered in detail by special libraries with distinct names for ucp/ucs/etc lengths, iterators, etc. There is no such thing as string length or "char" anymore. String is blank or non-blank, anything beyond that is too complex to be part of any stdlib. Even "blank" is not so obvious today.

ricardobeat 9y ago

'mañana'.padStart(7, '1') => "1mañana" // ok 'man\u0303ana'.padStart(7, '1') => "mañana" // oops You are being disingenuous here. Those are different strings, with different lengths (try copying this into the console): 'mañana'.length // 6 'mañana'.length // 7 The latter has two stacked characters. These issues are inherent to Unicode and `padStart` is treating the strings correctly. If you need normalization, use the .normalize method you mentioned yourself. This is a major improvement: double-wide and stacked characters have been there since ES3, but now the language is providing standard tools to work with them.

yarg 9y ago
I don't think that it's really fair to say that he's being disingenuous here, regardless of the underlying byte form the strings look indistinguisable and often users (devs) will expect them to function as such. I think it would be less confusing to define .length as the number of characters and have an additional .size method returning the number of bytes (I'm assuming that's what .length returns, if not it's even more confusing). Of course, that already wasn't done - meh.
- mbell 9y ago
  > have an additional .size method returning the number of bytes (I'm assuming that's what .length returns, if not it's even more confusing). It's actually not the number of bytes, it's the number of...'codepoint pieces' is what it could be called I guess? Javascript's language level string implementation is something like UCS-2 with the addition of surrogate pairs being allowed, but counted as separate 'characters' for things like length and index access. It's some twisted middle ground between UCS-2 and UTF-16.
  - yarg 9y ago
    That seems deranged to me. Like a true length calculation, It still requires a complex (albeit cachable) calculation to resolve, but it fails to return the length of the string in terms of the number of characters as they would be natually presented. I understand a need in some contexts to distinguish between a character and its subsequent modifiers - but I do not see such a context here. Design by committee?

tracker1 9y ago

That's why I hated the "it might break stuff" arguments against making "string" interfaces against characters (including combinators), and always using UTF-8 for encoding internally, in memory. Would have made a lot of that easier. As to your last bit, I tend to favor encodeURIComponent and have done it correctly... the main reason, is to avoid "+" vs " " in query strings.

lngnmn 9y ago

Holy fuck! The Fractal Of Bad Design all over again...