10 ms·
Has anyone here ever had a use case for toLower() where they actually wanted localization to apply? It seems to me that in practice, it's extremely rare to wan
by kentonv 6y ago
Has anyone here ever had a use case for toLower() where they actually wanted localization to apply?
It seems to me that in practice, it's extremely rare to want to change case of real, natural-language text. When I have natural-language text, it's just a blob to me, and I don't want to touch it.
The only time I ever want to lower-case or capitalize something, I'm working with identifiers meant for computer -- not human -- consumption. Usually, specifically, I'm dealing with identifiers that have annoyingly been defined to be case-insensitive even though the only humans that ever see them are programmers and programmers hate case-insensitivity. HTTP headers are a common example.
I mostly write C++, and I end up writing code like:
for (char& c: str) {
if ('A' <= c && c <= 'Z') c = c - 'A' + 'a';
}
Later on, some well-meaning developer on my team will come along and say "Ugh what is this NIH syndrome?" and then they "clean it up" as:
#include <ctype.h>
for (char& c: str) {
c = tolower(c);
}
And then I have to say NOOOOOOO DON'T DO THAT YOU HAVE NO IDEA WHAT tolower() REALLY DOES!
I struggle to imagine any real use case where you'd actually want locale-dependent tolower() other than, maybe, a word processor -- but if you're writing a word processor, you're probably not going to be depending on the language's built-in string APIs to do your text manipulation.
- rkangel 6y agoThis is a classic case of a 'why' code comment being needed. It's obvious what you're doing, but without a 2 line explanation, it's not clear why.
- kentonv 6y agoYeah I probably wrote that comment the first few times I did this but it's hard to write it the 50th time. Maybe I should have my own tolower() function that I can call so I only have to write the comment once but it just feels ridiculous somehow.
- Natsu 6y agoIt's far more ridiculous to repeat yourself over and over instead of making a simple function that describes exactly what you want and why.
- random314 6y agoWhy does it feel ridiculous?
- kentonv 6y agoBecause I've already rewritten more of the standard library than is healthy. I mean, it's clearly the right thing to do here but I can predict the conversation that will inevitably result... "You wrote your own tolower() function? Why?" "The standard one is horribly broken." "How could a function that lower-cases a letter be broken??? Jesus Kenton your NIH syndrome is out of control." "Sigh..." (Slightly more seriously, any particular time I need to lower-case something, it takes 10 seconds to write out the code, but would take 10 minutes to find a good place to define a reusable function and exactly what its API should be, and so it never seems worth the effort in the moment. Just like how most messy code comes to be.)
- nitrogen 6y agoMost codebases I've worked with have a StringUtils.java, or .kt, or a str.c or utils.c. Maybe just start one. Interestingly I haven't needed it as much in Ruby. But I too feel the cognitive (and social!) burden of introducing a new function. It's not just "where do I put this", but "how do I convince the team I know what I'm doing since 15 years of experience clearly isn't enough and developers (mostly rightly) ignore positional authority and seniority".
- random314 6y agoThis conversation can be simply be avoided by copy pasting your original hacker news comment into the library function header. I have noticed some coworkers have their ego gratified by being right while everyone else is wrong. Instead of simply explaining what they are doing when they are doing it, they will do something that looks wrong in a very noticeable way and wait for the backlash. The backlash gives them an opportunity to show everyone else how they were right while everyone else was wrong and also an opportunity to play victim. However, in SW development - it is not just the technical details - your behavior also matters in a big way. In this particular case, the correct approach is to create your own library function with appropriate comments. This is why the concept of a library function was invented. It is its entire raison-d'etre. However, you are doing everything but that. Including providing justifications in hacker news comments instead of your source code. Now inevitably, someone will change your inline code to use to_lower. This will give you an opportunity to scream bloody murder, show how other engineers don't really understand technical details, correct them and also play victim. Create a library utility with comments and link it in - End of story.
- eitland 6y agoWrite the function, comment it! Many of these are obvious to many people here, but some aren't. Even I can admit that some of the stuff in this thread is not obvious at all.
- Izkata 6y ago#include <kentonv.h>
- kentonv 6y agoIt's... It's called KJ... https://github.com/capnproto/capnproto/tree/master/c++/src/kj https://github.com/capnproto/capnproto/tree/master/c++/src/k...
- lathiat 6y agoThis project has the greatest sales pitch I've ever seen: https://github.com/capnproto/capnproto/blob/master/README.md https://github.com/capnproto/capnproto/blob/master/README.md -- combined with the project name it's just perfect.
- viraptor 6y ago> but it's hard to write it the 50th time. You know you wrote it 49 times before, but the person reading the code probably doesn't. It only changes your experience not everyone else's. If it's the same codebase, just write that function :)
- dmurray 6y agoSeems like it would be even better to put this in its own function with a descriptive name, ascii_tolower or roman_tolower or whatever, that has exactly the semantics you want.
- eitland 6y agoStill warrants a comment to be sure no one concludes the built in is good enough.
- randomdata 6y agoIt certainly warrants a test to document what the function is for. And if that test also happens to validate that the documentation is accurate, that is a nice side benefit.
- h0l0cube 6y ago> It certainly warrants a test to document what the function is for. You might be being serious, so I'll indulge. How would a test that will never break, for a function that would never be changed (because look at it), that lives in a different part of the directory structure, be worth even one second extra time or thought to write it, over-and-above the descriptive comment that is longer than the code itself, and lives discoverably right next to the code itself?
- xahrepap 6y agoI've found that writing a unit test to verify it works at all is sometimes loads easier than manually running the app or whatever. At that point, just leave the test and check it in.
- h0l0cube 6y agoI agree with this, kind of. Though I've found that keeping useless tests around doesn't always have 0 cost, same for any dead code. Often, though, you can verify that your code works using a REPL. I'm also a huge fan of how doc-tests in some languages that make the tests are part of the documentation, and neatly 'cohere' them to the function itself. At which point I'm happy to leave them in, as the tests-as-documentation are harder to miss, and are actually instructive.
- lmm 6y agoComments are unreliable, you should use tooling to fix this forever. E.g. findbugs has a rule for this problem: http://findbugs.sourceforge.net/bugDescriptions.html#DM_CONVERT_CASE http://findbugs.sourceforge.net/bugDescriptions.html#DM_CONV...
- grumple 6y agoNow I’m wondering about what happens when we change email addresses to lowercase... https://en.m.wikipedia.org/wiki/Email_address#Internationalization https://en.m.wikipedia.org/wiki/Email_address#Internationali...
- sedatk 6y agoYou shouldn't. Email addresses are case-sensitive.
- happytoexplain 6y agoWe frequently use localized upper/lower casing at my workplace, as we do not store such stylization in user facing copy. Most copy is written and translated in sentence case or title case (because both are much harder to achieve programmatically), and then our designers have the option of using that casing as-is, or using all-upper or all-lower.
- smnrchrds 6y ago> Has anyone here ever had a use case for toLower() where they actually wanted localization to apply? On many documents, including Turkish passport and identity card and many (all?) other passports, names are written in all caps. Maybe toLower() is not that useful, but toUpper() is crucial in any application where you are dealing with real person names.
- phonebanshee 6y agotoUpper is definitely language-dependent. For example, in Irish there are initial letters that are written as lower-case even in all caps. Wikipedia's example is amusing, since it's a photo of a government passport office sign - the all-caps version of Oifig na bPasanna is OIFIG NA bPASANNA (photo https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/ALLCAPS_OIFIG_NA_bPASANNA.JPG/440px-ALLCAPS_OIFIG_NA_bPASANNA.JPG https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/AL..., article https://en.wikipedia.org/wiki/Irish_orthography#Capitalisation https://en.wikipedia.org/wiki/Irish_orthography#Capitalisati...). It would look utterly bizarre to write OIFIG NA BPASANNA. And this isn't at all an unusual construction in Irish, it happens in personal names all the time. Plus, there's the issue of diacritical marks. Irish keeps long marks over capitals, but French drops accents. Do you plan to do é => É (required for Irish - POBLACHT NA hÉIREANN is the all-caps version of Poblacht na hÉireann [the Republic of Ireland]) or é => E (common practice for French)? You have to get it right, and you have to know the language to do that. (Poblacht na hÉireann also illustrates the fact that initial caps is also a language-dependent idea; you absolutely can't write Poblacht Na Héireann - that makes my eyes burn just looking at it.) (And before you say, well, Irish isn't a language spoken by very many people, remember that it's an official language of the European Union. If you're writing software to be used by EU agencies, you're going to have to care.)
- smnrchrds 6y ago> French drops accents Official position of both Académie française and Office québécois de la langue française is that accents must be preserved in capital letters. However, it is common in France to drop them, while they are almost always preserved in Québéc. I have heard that the reason is that European French keyboard layout makes it difficult to type accented capital letters, unlike Québécois French layout which makes writing them easy. But I am not sure if this is the cause rather than the effect of the practice.
- felixarba 6y agoI have a morse code app which consistently crashed when certain users would try to translate letter "i", and it took me a long time to figure out that only the turkish users would complain about it, and when one of them sent me a screenshot I only noticed a "wrongly" rendered capital letter i (I used toUpper) and after digging around a bunch, I learned about this while turkish letter i.
- fovc 6y agoWhat about sorting users by name?
- phonebanshee 6y agoThat's completely language+locale dependent. For example, here'an alphabetical list of Irish surnames - https://www.duchas.ie/en/nom?txt=M https://www.duchas.ie/en/nom?txt=M. You'll notice that sort order ignores an initial O or Mac (or Ni or Bean, etc).
- a-nikolaev 6y agoI think, this is why you need explicitly ASCII and explicitly Unicode lower-/upper-/capitalized-transformations. So you don't assume these things to work automagically. Some times you need one type, the other times you need the other type.
- deleted 6y ago[deleted]
- ramshorns 6y agoA char in C++ is one byte, right? Is it even possible for this "fixed" code to call ctype::tolower() on something like a UTF-8 or UTF-16 code point?
- kentonv 6y agoCorrect, it won't even work as intended with modern Unicode locales.
- ramshorns 6y agoSo maybe if the code is broken anyway for non-ASCII characters, it's fine to use tolower, since somewhere else in the code it ensures that c is a byte.
- kentonv 6y agoThe code is not broken for non-ASCII characters. UTF-8 works just fine with 8-bit chars, and the code I wrote correctly lower-cases ASCII letters even when UTF-8 is present (it just won't touch the UTF-8 chars, which is fine in this use case). It's only tolower() and toupper() specifically that are broken because they expect to be able to do their job on a single byte, which is no longer possible with UTF-8. Meanwhile, using tolower() to lower-case an HTTP header name won't give you the correct results if the locale is set to Turkish with the ISO 8859-9 character set, which is 8-bit, and where tolower('I') will produce the byte 0xFD which is 'ı' in this character set.
- ramshorns 6y agoI see, thanks for the explanation.
- cesarb 6y agoJava has two variants of toLowerCase(): one which uses the default/current locale (almost never what you want), and one which receives an explicit locale (Locale.ROOT is almost always the one you want). At work, we use the "forbidden APIs" checker (https://github.com/policeman-tools/forbidden-apis https://github.com/policeman-tools/forbidden-apis) to fail the CI if the variant which uses the default locale is ever used; if you really want to use a locale-dependent toLowerCase(), you have to explicitly call Locale.getDefault() and use it as the locale. Is there something similar for C and C++? It could help in your case, by making your well-meaning colleagues aware of the issue.
- rvnx 6y agoYes, tolower_l(string, locale)
- vesinisa 6y ago> Locale.ROOT is almost always the one you want At least Android developers are advised to use Locale.US: https://developer.android.com/reference/java/util/Locale https://developer.android.com/reference/java/util/Locale > The default locale is not appropriate for machine-readable output. The best choice there is usually Locale.US – this locale is guaranteed to be available on all devices, and the fact that it has no surprising special cases and is frequently used It would be indeed interesting to see in which features these two locales actually differ.
- nurettin 6y agoYou need localization if you do any kind of multilingual text processing. Not sure how it could escape a thinking person's imagination.
- jmiller099 6y agoi like c |= 0x20; :)
- tyingq 6y agoAirlines might be a good example. The back end system doesn't grok lowercase characters at all, so you need to transform data to uppercase A-Z, 0-9 and a few punctuation marks.
- miahi 6y agoBut they do have the most extensive transliteration rules library to match everything to that limited character set (ICAO Doc 9303[1]) that is used by many systems outside the aviation world. [1] https://www.icao.int/publications/pages/publication.aspx?docnum=9303 https://www.icao.int/publications/pages/publication.aspx?doc...
- __s 6y agoI recently ordered a Pixel, on the mail slip they had converted my name to uppercase, last name read "DUBé" Also got my address screwed up on account of living at a half address.. 1/2 some street #42
- aflag 6y agoFile names, URLs and email address support utf-8 characters and you may want to lower case them in many situations. If the user is trying to search for a string, they probably want case insensitivity. I don't think it's that rare/weird for people to want localisation to apply when calling toLower.
- reaperducer 6y agoHas anyone here ever had a use case for toLower() where they actually wanted localization to apply? Yes. In a system I'm about done with, there is a sortable chart of dates and times. In some languages day and month names are capitalized, and in some they are not.
- bjourne 6y agoHow does that work? toUpper() can't possibly know that the string is a day or month name.
- crazygringo 6y agoOf course! There are tons of cases where you need to store in "sentence case" (first word and proper nouns and acryonyms capitalized, nothing else) so you can convert to title case or all-caps as needed for display purposes. Templates are full of this kind of stuff. There are similarly tons of cases where you reduce everything to lowercase without accents for searching and indexing purposes. Depending on your setup, your database might handle that for you, but there are edge cases where you need to do it at the application level. Long story short, every string has a locale, and you should never change the case of something without specifying its locale. Either be explicit that it's American English or ASCII or Latin1 or whatever... or that it's something else. Never leave someone reading the code guessing.
- asveikau 6y ago> you can convert to title case or ... for display purposes. I am skeptical if someone thinks they need to do this and how they will get it done. Eg. Looping through and capitalizing the first gylph after breaking whitespace regardless of locale is not the way to go, but I guarantee you a nontrivial amount of people reading this would write exactly that if asked to solve the high level problem. I find it annoying when software or even in some cases human typists try to enforce English language title case. Some other languages have different rules for titles and capitalization and seeing the English rules enforced out of context can be jarring.
- crazygringo 6y agoI find it amusing you are skeptical... why so distrustful? But believe me, it's quite necessary. I use the citation manager Zotero a lot. It's necessary to store all the titles of journal articles and books in sentence case (e.g. "Issues regarding the economy of France") because some publications require citations to use sentence style (remains unchanged) while others require title style ("Issues Regarding the Economy of France"). And obviously the solution cannot be naive, but is language-dependent, so that in English words like "the" or "in" don't get capitalized. And as the rules for titles are obviously language-dependent, so it goes without saying that the algorithm would have to be localized. (Note that while it's relatively trivial to convert from sentence case to title case in English, it's impossible to automate in the opposite direction, because you never know if a capitalized term in the title is a proper noun or not.)
- deleted 6y ago[deleted]
- mrighele 6y ago> Has anyone here ever had a use case for toLower() where they actually wanted localization to apply? If you are collecting data which include people's names or addresses you probably want localization to be applied correctly so that you can compare data coming from different sources and possibly with different cases. Having your name spelled differently in different documents can cause a non trivial amount of problems with an overzealous bureaucracy.
- bjoli 6y agoI was onced refused entry into a country for 6 hours because of different spellings of my last name. The (apparently quite amateur) travel agency had sent my last name written using OE instead of Ö, whereas all the documents relating to my identity use Ö (or if it was the other way around).
- dusted 6y agoWe thought on this specifically when naming our son, one with no special characters, a pure clean ASCII string like mom used to cook them.
- vasama 6y agoThis is why I have a set of functions like AsciiToLower(char* string, size_t size). They only touch characters in the ASCII space at <0x80. Even went and implemented them with SSE for x86.
- mehrdadn 6y ago> Has anyone here ever had a use case for toLower() where they actually wanted localization to apply? How do you lowercase without localization? Remember all text isn't English. Unless you're actually asking if anyone has ever had a use case for lower-casing non-English text?
- lmm 6y agoI think the real question is: is there any use case for toLower() where you want the system default locale to be applied? If you want to lower-case text for "system" purposes then you need to keep track of the locale associated with that text (which won't generally be the locale of the system the program is running on); the only case where you want to use the system default locale is where you're interacting with the (human) user of the system, but it's hard to imagine a use case where you'd want to use toLower() for displaying text to the user.
- mehrdadn 6y ago> it's hard to imagine a use case where you'd want to use toLower() for displaying text to the user. Maybe for automatically changing the case of user input (auto-correcting capitalization, etc.).
- phonebanshee 6y agoYou vastly underestimate the desire of people for things to look 'nice' where 'nice' is defined by exactly what they mean when they say it. If you want a 'nice' display of data that's input by users, you're often going to butcher it by doing things like converting everything to lower-case, then maybe upper-casing the first letter. Because 'nice.' It's horrifically wrong, of course, but no one is going to think you're a reasonable person for insisting that correct wins over nice. Imagine a real-world scenario where you're displaying a list of names of users, where the users got to type in their own names. You can either use what users typed in, or you can do something like process it so it's in the American idea of initial caps. You can't possibly do localization, since it's a list of names of people in the US, so it's melting pot of names from all over the world, and you never asked for user input of what you'd use to localize it anyway [and no, you absolutely can't figure that out from just the name itself]. You can't use what users typed in, because the design team thinks that looks like a horrible mess (and it is; users are laughably bad at data entry). So the design team wins, and you butcher everything by pretending that toLower() plus toUpper() for the first character of every word is a sensible thing to do. (And yes, that's a painful real-world example of software I've shipped and that was used by millions of people.)
- karmakaze 6y agoAnd a note that it assumes ASCII. On an EBCDIC system, the 'A'-'Z' test will translate other characters besides letters.
- dragonwriter 6y ago> Has anyone here ever had a use case for toLower() where they actually wanted localization to apply? Well, it's always been exclusively in American English, but I've certainly used it in cases where I was doing text transforms for display, so, yeah, though it's not the most common case.
- neeeeees 6y agoI may be missing something - why is tolower(c) incorrect here?
- kentonv 6y agoBecause if `c` is the letter 'I', and the current locale happens to be set to Turkish, then `tolower(c)` will return 'ı', not 'i'. If you are trying to lower-case an HTTP header name for the purpose of case-insensitive comparison, this is definitely not what you wanted. (And similar problems exist with several other locales; it's not just Turkish.)
- neeeeees 6y agoAh I see, thanks for explaining
- bschwindHN 6y agoAhhh, mastahpiece https://doc.rust-lang.org/std/string/struct.String.html#method.make_ascii_lowercase https://doc.rust-lang.org/std/string/struct.String.html#meth...
- ezoe 6y agoA small portion of people whose native writing system are based from Latin alphabet believes that case conversion is an essential must have feature and having that feature in locale library help the localization. But if you consider the other writing systems, having case conversion feature in locale library actually harm the localization effort. It's not easy to make it no-op. The implementation of locale library are generally poor quality because the implementer has no idea how other languages works. Another example is singular/plural support. It just burden the localization effort because for languages which no such concept, the localization work must ensure that presence of such library doesn't harm their language. Some people is under the delusion that locale library must have more features to support his native languages not so important traits. While what really necessary is just forget about supporting minor language traits that are not universal among the languages. The text should be considered binary blob and most program should pass it left to right without modification.
- olliej 6y agoYes, semi regularly -- lowercasing of text for user interfaces is frequently required. Similarly case insensitive comparisons. Human text is much much more complex than any computing protocol you're ever going to engage with. The question is "which one should be default", and that's a more complicated question.
- golergka 6y agoHonestly, strings that are intended for human and for computer consumption should just be two different basic types without any implicit conversion between them.