6 ms·
> Except almost everyone always means #2. Until the string has to be stored in a database. Or transmitted over HTTP. Or copy-pasted in Windows running Autohotk
by dotancohen 5y ago
> Except almost everyone always means #2.
Until the string has to be stored in a database. Or transmitted over HTTP. Or copy-pasted in Windows running Autohotkey. Or stored in a logfile. Or used to authenticate. Or used to authorize. Or used by a human to self-identify. Or encoded. Or encrypted. Or used in an element on a web page. Or sent in an email to 12,000,000 users, some of whom might read it on a Windows 2000 box running Nutscrape. Or sent to a vendor in China. Or sent to a client in Israel. Or sent in an SMS message to 12,000,000 users, some of whom might read it on a Nokia 3310. Or sent to my exwife.
- arcticbull 5y agoOr sorted! That’s its own special hell.
- dotancohen 5y agoOr compared! How did I even forget about that. There is no form of normalization that covers all use cases. Full text search... Oh I want to cry...
- JohnHaugeland 5y agoEasy enough to normalize the terminology for length to mean character count and size to mean byte capacity.
- dotancohen 5y ago"I don't know or care what a character is. You're a character. Now limit this field to exactly 50 letters. Not 51. And not 49." And that client doesn't consider spaces nor punctuation as letters. Numbers count, though. Thankfully at the time emojis were not a consideration.
- arcticbull 5y agoSo you mean grapheme clusters or code points? Do you want to count zero-width characters? More specifically what are you trying to do?
- 3np 5y agoGrapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.
- AstralStorm 5y agoThat makes length operation require full parsing of the text according to Unicode rules. Hope you have an up to date set of these! Codepoints is easy, as are bytes. Graphemes much harder. So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)
- 3np 5y agoI’m really not an expert but is there anything particular with this one? Looking here the answer looks like “0” for all contexts to me. https://codepoints.net/U+2060 https://codepoints.net/U+2060 And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw). Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
- JohnHaugeland 5y ago> And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language that's literally the core premise of the original post
- 3np 5y ago...is it?
- dotancohen 5y ago> Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case. Here's one. How many characters is this: שלום And how about this: שַלוֹם Notice that the former is lacking the diacritics of the latter. Is that more characters? Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?
- Sharlin 5y ago"Character count" is not a well-defined concept because "character" is not a well-defined concept in general.
- zionic 5y agoIt absolutely is, unless you allow xenophilia to overcomplicate the issue. You don't need to completely bork your string implementation to support every language ever concocted by mankind. No one is coding in cuneiform.
- simonh 5y agoA bold decision, to use the term 'easy' in a discussion of unicode.
- simonh 5y agoHowever in this case it seems a justified one, having read your outstanding rebuttal to a sibling comment. Much appreciated.
- tn13 5y agoEnglish speaking world has developed intuition about strings due to ASCII which simply fails when it comes to Unicode and that basically explains a lot of these pitfalls. String length when defined #2 is also fairly complex when it comes to some languages such as Hindi. There are some symbols in Hindi which are not characters and can never exist as their own character but when placed next to a character they create a new character. So when you type them out on a keyboard you have to bit two keys but only one character will appear on screen. Unicode too represents this as two separate characters but for human eye it is one. त + या = त्या Following code will print 4 console.log("त्या".length);
- DemocracyFTW 5y ago"symbols in Hindi which are not characters and can never exist as their own character but when placed next to a character they create a new character" a.k.a. 'ligatures', as in f+f+i -> U+fb03 'ffi'
- nisegami 5y agoI would consider ligatures a text rendering concept, which allows for but is distinct from the linguistic concept described by GP. Edit: to further illustrate my point, in the ligatures I'm familiar with (including the ones in your link), the component characters exist standalone and can be used on their own, unlike GP's example.
- shoeffner 5y agoIn the example "Straße", the ß is, in fact, derived from an ancient ligature for sz. Old German fonts often had s as ſ, and z as ʒ. This ſʒ eventually became ß. We (completely?) lost ſ and ʒ over the years, but ß was here to stay. Its usage changed heavily over time (replacing ss instead of sz), I think for the last time in the 90s (https://en.wikipedia.org/wiki/German_orthography_reform_of_1996 https://en.wikipedia.org/wiki/German_orthography_reform_of_1...), where we changed when to use ß and when ss. So while we do replace ß with ss if we uppercase or have no ß available on the keyboard, no one would ever replace ß by sz (or even ſʒ) today, unless for artistic or traditional reasons. Many people uppercase ß with lowercase ß or, for various reasons, an uppercase B. I have yet to see a real world example of an uppercase ẞ, it does not seem to exist outside of the internet. For example, "Straße" could be seen capitalized in the wild as STRAßE, STRASSE, STRABE, with Unicode it could also be STRAẞE. It would not be capitalized with sz (STRASZE) or even ſʒ (STRAſƷE – there is no uppercase ſ) – at least not in Germany. In Austria, sz seeems to be an option. So, for most ligatures I would agree with you, but specifically ß is one of those ligatures I would call an outlier, at least in Germany. P.S.: Maybe the ampersand (&), which is derived from ligatures of the latin "et", has sometimes similar problems, alhough on a different level, since it replaces a whole word. However, I have seen it being used as part of "etc.", as in "&c." (https://en.wiktionary.org/wiki/%26c https://en.wiktionary.org/wiki/%26c.), so your point might also hold. P.P.S.: I wonder why the uppercasing in the original post did not use ẞ, but I guess it is because of the rules in https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.... (link taken from the feed). The wikipedia entry says we adopted the capital ẞ in 2017 (but it is part of unicode since 2008). It also states that the replacement SZ should be used if the meaning would otherwise get lost (e.g. "in Maßen" vs. "in Massen" would both be "IN MASSEN" but mean either "in moderate amounts" or "in masses", forcing the first to be capitalized as MASZEN). I doubt any programming language or library handles this. I would not have even handled it myself in a manual setting, as it is such an extreme edge case. And I when I read it, I would stumble over it.
- Dylan16807 5y agoExcept for SMS, #2 should work fine for all of those uses. And the reason you'd need a different measure for SMS is because SMS is bad. And half the things on your list are just places that Unicode itself might fail, regardless of what the length is? That has nothing to do with how you count, because the number you get won't matter.
- CodesInChaos 5y agoThe main reason I care about the length of strings is to limit storage/memory/transmission size. 1) and to a lesser degree 4) achieve that (max 4 utf8 bytes per codepoint). 2) comes with a log of complexity, so I'd only use 2) in a few places where string lengths need to be meaningful for humans and when stepping through a string in the UI. * One (extended) grapheme cluster can require a lot of storage (I'm not sure if it's even bounded at all), so it's unsuitable for length limitations * Needs knowledge of big and regularly updated unicode tables. So if you use it in an API chances are high that both sides will interpret it differently, so it's unsuitable for API use.
- Dylan16807 5y ago> One (extended) grapheme cluster can require a lot of storage (I'm not sure if it's even bounded at all), so it's unsuitable for length limitations You could take inspiration from one of the annexes and count no more than 31 code points together. > Needs knowledge of big and regularly updated unicode tables. So if you use it in an API chances are high that both sides will interpret it differently, so it's unsuitable for API use. It depends on what you're doing. That could cause problems, but so could counting bytes. Users will not be thrilled when their character limit varies wildly based on which characters they use. Computers aren't going to be thrilled either when replacing one character with another changes the length. Or, you know, when they can't uppercase a string.
- jasode 5y ago>, #2 should work fine for all of those uses. No, #2-length is a higher-level of abstraction to display for humans. E.g. calculating pixel widths for sizing a column in a GUI table, etc. I don't think you understood the lower-level of abstraction in the examples that gp (dotancohen) laid out. (Encrypting/compressing/transmitting/etc strings.) In those cases, you have to know exact count of bytes and not the composed graphemes that collapse to a human visible "length". In other words, the following example code to allocate a text buffer array would be wrong: char *t = malloc(length_type_2); // buggy code with an undersized buffer