17 ms·
TIL the assumption that string length does not change when upper-cased is false
- bgro 5y agoThis sounds like a question they'd ask during a tech interview and when you ask which language or what they mean by "length," the "senior" developer says, "Heh, guess you're not one of us who work for a living. You couldn't even answer a basic programming question. You see, length means how long a thing is. Everyone knows that. NEXT."
- darepublic 5y agoProbably would have assumed the same. Though thinking about it harder and remembering i18n should provide a clue
- TimTheTinker 5y agoThis is why the `toLocaleUpperCase` method exists (also `toLocaleLowerCase`). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleUpperCase https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- gpmcadam 5y agoSee also: `String.prototype.localeCompare` https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- netcraft 5y agoI sort of expect that nothing can be assumed when talking about strings and characters anymore. Waiting for the post on HN one day that says that its in the unicode spec that characters can animate or be in superposition until observed…
- tshaddox 5y agoOr that some people’s names can only be represented as a bitmap or vector graphic. Or what if some people’s names can only be represented in a computer by that computer executing arbitrary code? Then all computer software that accepts the input of human names must by definition have arbitrary code execution vulnerabilities!
- colanderman 5y ago> unicode spec that characters can [...] be in superposition until observed You mean like language-dependent unified Han glyphs? [1] [1] https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
- junon 5y agoWhen I read GP that was the first thing I thought of. Animated emojis probably aren't too far off the horizon, and I could definitely see a world where multiple glyphs could be combined to affect the animation much like we do for skin tone or the family emojis. Sometimes I think we've gone too far.
- phkahler 5y ago>> Sometimes I think we've gone too far. Agreed from a technical standpoint. But also dont forget we have the poop emoji and the love-hotel.
- majou 5y agoThat's just its Japanese origin.
- jchw 5y agoEmojis in Unicode may have been massive scope creep, but I don’t think the poop emoji or love hotel emoji represent it. Love hotels are a legitimate phenomenon in Japan, where emoji originate (heck, even the word emoji is Japanese.) It fits in well alongside other map icons, even if the cultural clashes can be amusing. And the poop emoji may be silly, but it’s hard to deny its application for expression. Now what I find is less practical and more ridiculous is the sheer number of ways you can combine various codepoints to get different emoji representing families. On one hand, it’s impressive dedication to trying to be neutral and encompassing, on the other hand it is nightmarish just how many possibilities exist.
- euske 5y agoI just found that this applies to Python too. >>> 'Straße'.upper() 'STRASSE'
- hulahoof 5y agoNow this has me wondering how you would be able to reverse something like this to get something like: >>> 'Straße'.upper().lower() 'straße' instead of: >>> 'Straße'.upper().lower() 'strasse'
- pvg 5y agoThe conversion is lossy since lowercase ss (e.g Gasse) is valid and common German and capitalizes to SS as well. Short of the call learning some German, there's no neatly reversing this.
- ThreeFx 5y agoThere are even cases where it is context-depedent: Maße (measurements) and Masse (mass). Both are written MASSE uppercased, but how do you know which one's which? For additional fun: Swiss German doesn't use ß at all, so there is no difference between "in Maßen genießen" (to enjoy in moderation) and "in Massen genießen" (to enjoy in large amounts).
- pvg 5y agoSwiss German doesn't use ß at all TIL! I googled my way to a fun summary of the various factors potentially influencing this: https://german.stackexchange.com/questions/56567/why-was-%C3%9F-discontinued-in-switzerland https://german.stackexchange.com/questions/56567/why-was-%C3...
- detaro 5y agoyou can't
- naniwaduni 5y agoYou can't without out-of-band information.
- cgtyoder 5y agoIt's a bug. There is a capital eszett now, as of 2017. https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F
- foepys 5y agoIt's not a bug. It's a legacy artifact that cannot be changed easily because of backwards compatibility.
- dredmorbius 5y agoAs noted in the discussion, this isn't the sole case. There are also non-invariant case-modifying examples, where uppercase -> lowercase -> uppercase (or l -> u -> l) does not produce the input string.
- andi999 5y agoIf you read the official announcement for the German language (cited in the wiki article) there it states that SS is the capitalization of ß, but ẞ instead is permitted. So the standard Form is double S.
- haimez 5y agoQuestion: is there any reason to expect that the double s form will be replaced, or is this a more of a permanent quirk of a language adopting quirks from their neighbors?
- andi999 5y agoDifficult to say. The double s Form has to stay (for small and capital) to not deviate from Swiss German too much (has to stay compatible). Generally older people think capital ẞ is ugly, but maybe this will change over generations. Best bet (for me) is that all ß fall out of favor and Germany goes the Swiss way. Let's talk in 60 years again.
- iggldiggl 5y ago
- arcticbull 5y agoThe issue is that the concept of “length” without additional context no longer applies to Unicode strings. There’s not one length there’s four or five - at least. 1. Length in the context of storage or transmission is the number of bytes of its UTF-8 representation - and not just that, one of its decomposed forms too. 2. Length in terms of number of visible characters is the number of grapheme clusters. 3. Length in terms of visible space on screen is the number of pixels wide and tall when rendered in a given font. 4. Length when parsing is based on the number of code points. If you look at it that way it makes perfect sense that the “length” of a string could change when any operation is performed on it - because there’s no such thing as a canonical length. tl;dr bug or not, the concept of a canonical string length is an anachronism from the pre-Unicode days when all those were the same thing. There’s no such thing as a string length anymore.
- darrenf 5y agoRaku provides explicit methods for #1, #2 and #4 on strings: $string.encode('UTF-8').bytes; #1 $string.chars; #2 $string.codes; #4 https://course.raku.org/essentials/strings/string-length/ https://course.raku.org/essentials/strings/string-length/
- goldenkey 5y agoFor those that don't know, Raku is just Perl 6 renamed.
- lizmat 5y ago> is just Perl 6 renamed Though this was technically correct in 2019, meanwhile the Raku Programming Language has continued to evolve, de-emphasizing its heritage more and more. For example, Covid restrictions allowing, 2022 will see the first European Raku Conference.
- goldenkey 5y agoIs it a fork..or just a rename for a breaking version tree? I understand that backwards compatibility is not a required goal. So isn't it more like Python 3 rather than an entirely different programming language?
- varenc 5y agoAnother fun fact: Upper-casing is language dependent. In English uppercasing 'i' gets you 'I'. But Turkish has a dotted and un-dotted 'i', each with an uppercase variant. So if your user's language was Turkish, uppercasing 'i' would give you 'İ', and lowercasing 'I' would give you 'ı'. Makes me wonder how case insensitive file systems handle this...and for more fun, handle the situation where the user changes the system language. I know that the Turkish 'I' delayed at least one big company's Turkish localization efforts for awhile.
- toast0 5y agoMy understanding is that case insensitive filesystems that wish to be portable have a mapping table in the metadata. A quick search showed references to an 'upcase table', although I'm not sure of the accuracy of the source, so I won't link it. Just because the user changed the system language doesn't mean the system should be expected to change the upcase table though. That operation would need to be very carefully managed; you can't change the rules if there are existing files that would conflict in the new mapping. And you might have symbolic links that matched because of case insensitivity that won't anymore... Pretty tricky.
- DemocracyFTW 5y agoI think this goes to show that the file system is not the right place to implement natural language handling to its full extent, like it's probably not a good idea to implement time zones as a primitive function in your new programming language.
- rini17 5y agoWhat is the right place to have two different systems agree on natural language handling, then? And of course there are widely recognized problems with time zones too.
- lifthrasiir 5y ago> Makes me wonder how case insensitive file systems handle this... They generally don't. It is true that several case-insensitive file systems (including NTFS, exFAT and ext4 [1]) maintain some sort of configurable case-folding maps but they are mostly used to guard against the future Unicode update and do not vary across locales. [1] https://dfir.ru/2021/07/15/playing-with-case-insensitive-file-names/ https://dfir.ru/2021/07/15/playing-with-case-insensitive-fil... (ext4 supports an optional case insensitivity)
- sillysaurusx 5y agoI'll paypal $20 to anyone who can name a situation where string length (in number of visible characters) is actually required by any reasonable algorithm. (High roller, I know.) No games, though. If you say "An algorithm to compute the length of the string in number of visible characters," that obviously is designed to pass the test rather than to do anything useful. Maybe framing it as a bet will break the illusion that string length is ever required.
- latch 5y agoI'm not sure I understand what you're getting at. But if changing the case didn't alter the length, you could do it in-place, without having to allocate memory. I always thought this was the point?
- toast0 5y agoVisible character count isn't necessarily related to storage size. If uppercase requires a diacritic, but lower case didn't that doesn't (usually) increase the visible character count, but it likely increases storage space, depending on encodings and normalization forms. Thinking about i to I with dot (which I can't currently type), which would need more storage in utf-8 or as I + dot.
- deleted 5y ago[deleted]
- yrral 5y agoWhen you have fixed width font and need to size columns appropriately for the data displayed in eg: a table.
- lofties 5y agoOh that's a great one! I have a thermal receipt printer (can recommend anyone to buy one, they're amazing) that is 80 characters wide. Would definitely screw up the formatting if it ever encountered this situation. (and no further validation was performed on the uppercase'd string)
- heavyset_go 5y agoPython has str.casefold() for caseless comparisons that handles the example in the OP[1]: > str.casefold() > Return a casefolded copy of the string. Casefolded strings may be used for caseless matching. > Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string. For example, the German lowercase letter 'ß' is equivalent to "ss". Since it is already lowercase, lower() would do nothing to 'ß'; casefold() converts it to "ss". > The casefolding algorithm is described in section 3.13 of the Unicode Standard. [1] https://docs.python.org/3/library/stdtypes.html#str.casefold https://docs.python.org/3/library/stdtypes.html#str.casefold
- mjn 5y agoI believe this does work for German (or at least I can't think of an example where it doesn't). But a case I can think of where it doesn't work is with standard modern Greek. In all-caps words, accents are omitted, while at least Python's implementation of casefold produces non-equal strings for the all-caps and lowercase versions: >>> str.casefold("παράδειγμα") 'παράδειγμα' >>> str.casefold("ΠΑΡΑΔΕΙΓΜΑ") 'παραδειγμα' It's at least consistent with this, though: >>> str.upper("παράδειγμα") 'ΠΑΡΆΔΕΙΓΜΑ' But I'd consider that incorrect, or at least nonstandard. The Greek alphabet does have accented versions of capital letters, but they can only be used as the first letter of a word in mixed case (e.g. if a sentence starts with έλα, you write it Έλα), never in the middle of a capitalized word. However maybe this slides too far to the "language" rather than "encoding" side of the space that Unicode considers outside of its purview.
- zvr 5y agoYes, the Python standard library (like many other implementations) chose to do the "easy" way instead of the "correct" one. Correct case-folding in Greek is complicated, since it might introduce diaeresis in the next vowel, if the accents are removed: Μάιος - ΜΑΪΟΣ. All this means that correct handling is not reversible, which introduces a slew of other problems.
- gfxgirl 5y agoSometimes I wish the popular programming languages had been invented by people who's written language had no concept of UPPER and lower case. There's so much cruft in code bases because of it. A conventions include kUPPERCASE for constants, CapitalizedCamelCase for classes and or functions, sometimes snake_case for variables or whatever. So then you have millions of wasted person hours of things that could have been automated if they matched, but they don't Example enum Commands { kCOPY, kCUT, kPASTE, } class CopyCmd : Public Cmd { ... } class CutCmd : Public Cmd { ... } class PasteCmd : Public Cmd { ... } Cmd* MakeCommand(Command cmd) { switch (cmd) { case kCut: return new CutCmd(); case kCopy: return new CopyCmd(); case kPaste: return new PasteCmd(); ... The fact that in some places it's COPY, in others Copy, and in probably others copy means work that would disappear if it was just always COPY. All of it superfluous work that someone from a language that doesn't have the concept of upper and lower case would never had even considered when come up with coding standards. Hey, I could just copy and paste this.... oh, the cases need to be fixed. Oh, I could code generate this with a macro.... oh, well, I've got to put every case form of the name in the macro so I can use the correct one, etc...
- BoorishBears 5y agoI don't get this... a) Is your tooling not identifier aware? That removes the hassle from remembering the case correctly. And for the matter so does having a consistent standard... b) When is capitalization ever the thing stopping you from copying and pasting some code? Capitalization only differs if the semantic meaning differs. Like what's a specific example in your code where this is avoided: > Hey, I could just copy and paste this.... oh, the cases need to be fixed. Oh, I could code generate this with a macro.... oh, well, I've got to put every case form of the name in the macro so I can use the correct one, etc... when all the capitalization matches?
- gfxgirl 5y agoI'm not sure what your point was. My point is because these languages were invented in English and because English language has a concept UPPER and lower case, English programmers made up naming conventions using different case variations of the same id. The most common convention I've run into I listed above. So I type out some enum using kUPPER for 50 things, I then copy and paste that somewhere to generate 50 functions, but according to naming conventions, I now have to fix the case of all of those functions. That "fix the case" is wasted work. Then we also end up with kUPPER except if it's a compound word so kPASTEANDGO becomes kPASTE_AND_GO which then has to be translated to some `class PasteAndGo` because of naming conventions. Again, wasted work. There are lots of cases where this conversion ends up causing all kinds of headaches, major and minor, just to keep everything matching the naming convention. Sure, you could choose to ignore the convention in this case but I've rarely seen a code base that does. Whereas, if the languages had come from a culture with the language that doesn't have the concept of UPPER and lower case then these conventions might never have happened. You'd have maybe something like K_COPY_AND_GO, CLASS C_COPY_AND_GO, etc and copying and pasting ids around or code generating there'd be no hoops to jump through to match the case conventions for different uses of the same id. > Capitalization only differs if the semantic meaning differs Yes, and I'm positing that prefixes or suffixes would be better than case differences
- avgcorrection 5y agoThis is an unreasonable expectation to have for Unicode anyway. 1. Assume that some letter X has only a lower-case version. It’s represented with two bytes in UTF-8. 2. A capitalized version is added way later 3. There are no more two-byte codepoints available 4. So it has to use three bytes or more I see people are jumping on the “oh the wicked complexity” bandwagon in here but I don’t see what the big deal is.
- ironmagma 5y agoPresumably string length corresponds to something like “number of glyphs” rather than byte count.
- sokoloff 5y agoæ, œ, and fi are one or two? They’re one glyph, obviously, but when you ask ‘how long is “ fish “?’, it would seem odd to answer “3”.
- DemocracyFTW 5y agoTo make things worse (or better), æ, œ, and ß have evolved from ligatures to becoming letters in their own right (a choice that may depend on the language), while fi stayed a ligature. So 'fieß', if there was such a word in German, would count as 4 letters. Also, talking about 'glyphs', one has to clearly separate between what Unicode encodes and what a given font uses; in the context of fonts, 'glyphs' are also called 'glyfs' or 'outlines', and any OTF my choose to render any given string with any number of glyfs/outlines, including using a visual ligature to display 'f+i' = fi but doing so by nudging the outlines for f and i closely together.
- ironmagma 5y agoI have no expectations anymore with Unicode, so I’m not surprised at all.
- vidarh 5y agoInterestingly, and I can't decide if I hate it or not, when searching for "æ" to get back to this comment after stupidly scrolling away from it, Chrome matched on "ae" in other comments, not just on the letter itself. In Norwegian, "ae" is an accepted transliteration of "æ", but "æ" is very clearly considered it's own character and we'd expect the answer to how many characters is "æ" to be 1 (which creates fun sorting issues too - "ae" is expected to sort after "z"), just like "aa" is a transliteration of "å" except when it isn't (it's always a transliteration when outside of names; within names all bets are off - some names traditionally uses "aa", some uses "å" and if represented with "aa" it respresents a transliteration; to make it worse, some names exist in both forms - e.g. Haakon vs. Håkon). Now the interesting question to ask Norwegians is "how many characters are 'ae'?" On one hand the obvious answer is 2, but I might pause to think "hold on, you meant to write "æ" but transliterated it, and the answer should be 1". Except it might occur to me it's a trick question - but it could be a trick question expecting either answer. Argh. [I just now realised a search for "å" matches "a" and "å", but "aa" is treated as two matches on "a", and that I definitively hate, though I understand it's a usability issue that it matches on "a", and that matching on "aa" makes no sense if the matched term is in another language.] [EDIT2: I've also done some genealogy recently, and to be honest, the spelling of priests makes it quite hard to hold on to any desire for precision in find/search]
- ir193 5y agoCase distinction does not happen in every language. I'm not sure if it's European-languages-specific.
- rvense 5y agoThe Latin, Greek and Cyrillic alphabets have case. I think that's it.
- JohnHaugeland 5y agoArmenian, Coptic, Cherokee, Garay, Osage, Adlam, Warang Citi, and Deseret, as well. These are called "bicameral scripts." Whether Japanese has them or not is arguable - consider katakana, where they switch for externality. Whether Arabic has it or not is also arguable. Many Arabic letters have alternate forms (English speakers might comparatively call them "modern," "holy", "cursive," and "historic.") You might argue that English has other alternate cases, such as cursive, handwriting, blackletter, doublestrike, smallcaps, title case, and so forth.
- 3np 5y agoHiragana/Katakana absolutely can not be interpreted as “uppercase/lowercase”. BTW kana also has full-width/half-width versions, which while also distinct would be closer. For a more pure example, consider Bopomofo. https://en.wikipedia.org/wiki/Bopomofo https://en.wikipedia.org/wiki/Bopomofo
- JohnHaugeland 5y ago> Hiragana/Katakana absolutely can not be interpreted as “uppercase/lowercase”. I didn't say it was upper or lower case. I said it was a difficult and arguable parallel, because just like upper and lower case, it's a setup where the letters you use are switched out for others because of context. . > For a more pure example, consider Bopomofo. From my perspective, it's literally exactly the same thing: a case where the letters you use are switched out because of context. In fact, my linguistics textbook gives Bopomofo as an example *in the same sentence as katakana*. Pardon me if I trust the linguists on this.
- spullara 5y agoWhat would have happened if Microsoft had decided they were only going to support ASCII?
- BlueTemplar 5y agoThey might have lost to IBM, who already had their own "Extended ASCII" ?
- AtlasLion 5y agoI was testing this lately,Teradata, SQL server, Oracle and others just return ß for upper('ß'). Snowflake returns SS. There is btw an upper letter defined for ß https://m.wikidata.org/wiki/Q9693 https://m.wikidata.org/wiki/Q9693
- eesmith 5y agoAlso, ligatures ("ffi" -> "FFI"). Here's the 102 single Unicode points which, after upper(), map to more than one point: >>> for i in range(1_114_112): ... s = chr(i) ... if len(s) != len(s.upper()): print(i, s, s.upper()) ... 223 ß SS 329 ʼn ʼN 496 ǰ J̌ 912 ΐ Ϊ́ 944 ΰ Ϋ́ 1415 և ԵՒ 7830 ẖ H̱ ... 8188 ῼ ΩΙ 64256 ff FF 64257 fi FI ... 64279 ﬗ ՄԽ
- OisinMoran 5y agoAs heavyset_go mentioned casefolding is your friend here, but you'll also likely want to do some unidecoding too depending on your use case. With that lens, going from one to two characters is pretty tame as there are numerous examples of single characters that unidecode to eight! And also ones that go to zero. We've done some pretty fun work at Inscribe ensuring things like this work across multiple languages, in particular for matching names. So we can now verify that a document does indeed belong to "Vladimir" even if the name on the document is "Владимир". The trickier part is not simply matching those, but returning the original raw result, as for that you need to keep track of how your normalization has changed the length of things. If you're interested in this kind of thing, shoot me a mail at oisin@inscribe.ai. We're growing rapidly and hiring to match.