6 ms·
> To disable slow i18n routines and use traditional byte-based sort order, use export LC_ALL=C (in fact, consider putting this in your ~/.bashrc). Do not. Havi
by scrollaway 11y ago
> To disable slow i18n routines and use traditional byte-based sort order, use export LC_ALL=C (in fact, consider putting this in your ~/.bashrc).
Do not. Having a non-utf8 locale means you won't be able to handle utf-8 sanely ("that's why it's faster") and it will break at the most inexplicable times. Any non-latin1 character appearing in your prompt or command line with this will mess its spacing up for example. Do not do not do not.
Hell I even check for it in my .zshrc: https://github.com/jleclanche/dotfiles/blob/master/.zshrc#L378-L381 https://github.com/jleclanche/dotfiles/blob/master/.zshrc#L3...
Good post otherwise.
- binarycrusader 11y agoI have to reiterate this; don't do it -- sort order changes and you'll run into other mysterious issues. It's not worth the "performance improvement".
- cle 11y agoIn some cases, it certainly is worth the improvement. Performance improvements translate into $$$. Knowing that "LC_ALL=C can improve performance but cause UTF problems" is awesome information to base a design on. A blanket statement like "never use LC_ALL=C" is not.
- binarycrusader 11y agoShort of a bug or other pathological issue, if you resort to messing with locale just to avoid a performance problem, you're likely to hide other issues that will cause problems whenever another user attempts to do the same thing and it breaks. Again, not worth it -- if only for performance sake, generally speaking.
- unhammer 11y agoThere are good reasons to use LC_ALL=C for certain commands, especially if you want sorting that's stable across different systems. Sometimes you want "aa" to be before "b" even though the Norwegian UTF-8 locale puts it after "å" (which comes after "…zæø"): $ echo $'a\naa\nb\nå'| LC_ALL=C sort a aa b å $ echo $'a\naa\nb\nå'| LC_ALL=nn_NO.UTF-8 sort a b å aa Even worse, some characters that are byte-different collate the same: $ echo $'∨\n∧'|LC_ALL=C sort -u ∧ ∨ $ echo $'∨\n∧'|sort -u ∨ Handling Unicode sanely requires understanding some Unicode.
- binarycrusader 11y agoI'm well aware of that, but in general, don't do it still applies :-) In general, I'd never do it for performance only, it would have to involve one of the issues you've pointed out (there are some components I build that require it, erroneously IMO).
- thristian 11y agoSome environments support a locale named "C.UTF-8" - with the same familiar sorting and formatting rules as the C locale, but with proper UTF-8 support.
- zalzal 11y agoUsing C locale is useful when processing very large data files in a shell, e.g. on a server. For example, uniquing or set difference will work many times faster (and doesn't require the slow/fancy collation calculations). But yes, it's a good point, of course it shouldn't be used on your desktop globally or apps will break. I'll clarify that.
- cmrx64 11y agoSetting LC_COLLATE=C and leaving the rest to a proper locale can often win you what you want without losing much.
- stass 11y agoThe question is, do you need to handle UTF-8? In most cases of sever management having LATIN1 would suffice, or appropriate localized 8-byte encoding. You can apply the same argument to any other obscure encoding scheme, not just UTF-8. The answer is really to pick the encoding which will suite the most usecases and won't be painful to use.
- gchpaco 11y agoHow is ASCII or Latin-1 somehow not an obscure encoding scheme by your parlance? Latin-1 is less common by quite a bit on the web than UTF-8, and ASCII might be if UTF-8 wasn't a superset. Fact of the matter is people have to get out of the habit of going "oh, that will never be relevant for me", because a) that's unlikely to be true for everybody (for example in a server environment you can't even write perfectly good North American names like Étienne let alone perfectly good older-than-the-country names like Þórr) b) it's unlikely to be true even for you as you will discover the first time you try to read a file written by somebody else c) it promotes intellectual laziness that makes later internationalization efforts difficult to impossible, because they're trying to convince people like you to care about stuff they find inconvenient but that is of vital importance to your customers. I know of a man whose first name is Þórr who outright refuses to deal with any organization that will not let him write his name, and I have a hard time disagreeing with him.
- stass 11y agoThe original question was about the command line. When managing servers it's perfectly accessible to use LATIN1 if one does not expect to deal with UTF-8 data (examining processes, VM states, changing configuration, etc). If that's not the case, one can use UTF-8 or some other encoding for that matter (i.e. I routinely have to deal with non-UTF non-ASCII filenames and text, using UTF-8 won't cut here either).
- scrollaway 11y ago> if one does not expect to deal with UTF-8 data This is a horrible assumption. UTF8 can show up just about anywhere. 99% of the utilities mentioned in the article support UTF8, which means they will output UTF8 if they think that's a good idea. And that can show up at any point. Exotic filenames, error messages, fancy boxing or indicative characters in the output... Don't expect Latin-1, ever. Respect specified encodings, and default to UTF8. Never, ever think you'll be fine with Latin-1 when dealing with an UTF-8 output just because you speak english. The trick the original article specified is a hack to get better performance in very controlled situations, it should never be put in a rc file like it recommends because it will break and you won't know that is what's causing it. Edit: Here's a few examples of very real potential breakage, to show you how bad an idea this is: You're debugging a python script. You open a python shell and start pasting some of its lines to figure out what's happening. OOPS, there's UTF-8 in there. [4:08:48] adys@azura ~ % python -c 'print("I like the letter Š")' I like the letter Š [4:08:54] adys@azura ~ % LANG=C python -c 'print("I like the letter Š")' Unable to decode the command from the command line: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc5' in position 25: surrogates not allowed (The input doesn't even have to be explicit, by the way, it could even be in your filenames). Should I keep going? See how "awk '{print tolower($0)}'" behaves when it's dealing with utf-8 input. Maybe that awk is deeply hidden in a 5000 line script. Maybe it deals with web data, filenames, anything. And suddenly you have unicode escapes in data that doesn't expect any. Now how's that one line in your rc file, which you wrote maybe months or years ago, working out for you? Are you going to know the issue is your LC_ALL variable?
- xenophonf 11y agoI have the following set in my .cshrc: setenv LC_ALL en_US.UTF-8 setenv LC_COLLATE C # use the ASCII sort order What have I broken, and how badly?
- scrollaway 11y agoYou're not breaking anything on a technical level with that. Be wary of anything that deals with string or character comparison however (including regexes).
- lejoko 11y agoThis won't work because LC_ALL overrides LC_COLLATE (and all the LC_* variables actually). It's LANG that you want to use. LANG is overriden by LC_* which are in turn overriden by LC_ALL. Don't use LC_ALL in your config files, use it only for one shot command lines or in scripts that need to be sure of the environment they run in.
- xenophonf 11y agoThank you both for the clue. I used to have LANG set, but I'm just going to unset them all. I think I put them in originally to change how directory listings got sorted, but I'm not sure I care any more. :)
- voltagex_ 11y agoRaised https://github.com/jlevy/the-art-of-command-line/issues/22 https://github.com/jlevy/the-art-of-command-line/issues/22
- scrollaway 11y agoThanks! Didn't think of filing it as an issue.
- zalzal 11y agoThe discussion here has tended to focus on either one or the other of two different facts: You absolutely need to be aware of and manage language and locale settings diligently. This is a big topic, but generally, whenever possible, use UTF8. And you generally don't want to change your desktop OS or applications' locale or language settings other than that. However, if you are processing data files and are aware of the implications, and can get away with treating data as binary instead of character streams -- for example, any of the command sequences that use sort/uniq for uniqueness or set union/intersection/difference -- then using C locale is way faster. As in, the difference between doing something in an hour on a big machine and rewriting a whole pipeline in Hadoop. A good fact to know. It's obvious I (the author) didn't make the simultaneous truth of both these points clear. I'll rewrite.