4 ms·

This is what I get on a Precise 32bit Ubuntu: stuff$ du -sh big.log 2.8G big.log stuff$ time grep -i e big.log > /dev/null real 0m30.228s user 0m12.

by iagooar 13y ago

This is what I get on a Precise 32bit Ubuntu: stuff$ du -sh big.log 2.8G big.log stuff$ time grep -i e big.log > /dev/null real 0m30.228s user 0m12.213s sys 0m3.228s stuff$ time LANG=C grep -i e big.log > /dev/null real 0m30.130s user 0m12.105s sys 0m3.308s What is LANG=C supposed to do?

pmelendez 13y ago
>What is LANG=C supposed to do? It changes the charset to do not use utf-8.
- ttflee 13y ago
  Not supporting Unicode often drive me crazy/sad, especially for a diff/patch tool.
  - justincormack 13y ago
    It doesn't actually change the charset, it changes the behaviour of a few libc functions and other things that use it (like perl). Sane programs ignore libc locales, as you do not want the behaviour of your program changing randomly based on some environment variables, that way madness lies.
  - UNIXgod 13y ago
    Having control makes me happy/sane. Having unicode set to (on|off) does not affect either diff/patch.
masklinn 13y ago
Sets the process's locale to C rather than whatever the system's default is (if the process is locale-aware, as "C" is the default locale for C programs). The main change[0] for this case is to also disable encoding (and thus decoding), all text is considered to be ASCII rather than whatever the locale specifies (usually UTF8 these days) [0] the locale should have an impact on what character ranges match. See http://stackoverflow.com/questions/6799872/how-to-make-grep-a-z-independent-of-locale http://stackoverflow.com/questions/6799872/how-to-make-grep-... for an example
simias 13y ago
Maybe your default locale is already C? I'm still surprised that TFA can claim such a speedup, I would have thought IO speed was the bottleneck when you grep through a big amount of data. As an other poster mentioned I wonder if the speedup is not mainly disk caching in RAM during the 2nd run.
- masklinn 13y ago
  FWIW my locale is en_GB.utf-8 and I also get no difference (in fact, the version with locale is slightly faster than without), with GNU grep 2.14 on OSX 10.9. The built-in BSD grep (2.5.1-FreeBSD) also runs in 30% of the time GNU grep does.
- a8da6b0c91d 13y ago
  I just did 'strings /dev/urandom > stuff' for 100M. Got it in memory first with untimed grep, then: $ time grep -i blah stuff bLAH blAH real 0m7.227s user 0m7.194s sys 0m0.011s $ time LANG=C grep -i blah stuff bLAH blAH real 0m0.486s user 0m0.467s sys 0m0.016s So a pretty big difference. My default lang is en_US.utf8.
UNIXgod 13y ago
It's historical and sets your local character set. The default value on FreeBSD is POSIX which is an alias the historically value which is C. Desktop unix's like OSX or ubuntu set it to utf8. Another note is that this is triggering fgrep which is already fast due to it's fixed length expression (i.e. no recursion is involved)

by iagooar 13y ago

pmelendez 13y ago

>What is LANG=C supposed to do? It changes the charset to do not use utf-8.

ttflee 13y ago
Not supporting Unicode often drive me crazy/sad, especially for a diff/patch tool.
- justincormack 13y ago
  It doesn't actually change the charset, it changes the behaviour of a few libc functions and other things that use it (like perl). Sane programs ignore libc locales, as you do not want the behaviour of your program changing randomly based on some environment variables, that way madness lies.
- UNIXgod 13y ago
  Having control makes me happy/sane. Having unicode set to (on|off) does not affect either diff/patch.

masklinn 13y ago

Sets the process's locale to C rather than whatever the system's default is (if the process is locale-aware, as "C" is the default locale for C programs). The main change[0] for this case is to also disable encoding (and thus decoding), all text is considered to be ASCII rather than whatever the locale specifies (usually UTF8 these days) [0] the locale should have an impact on what character ranges match. See http://stackoverflow.com/questions/6799872/how-to-make-grep-a-z-independent-of-locale http://stackoverflow.com/questions/6799872/how-to-make-grep-... for an example

simias 13y ago

Maybe your default locale is already C? I'm still surprised that TFA can claim such a speedup, I would have thought IO speed was the bottleneck when you grep through a big amount of data. As an other poster mentioned I wonder if the speedup is not mainly disk caching in RAM during the 2nd run.

masklinn 13y ago
FWIW my locale is en_GB.utf-8 and I also get no difference (in fact, the version with locale is slightly faster than without), with GNU grep 2.14 on OSX 10.9. The built-in BSD grep (2.5.1-FreeBSD) also runs in 30% of the time GNU grep does.
a8da6b0c91d 13y ago
I just did 'strings /dev/urandom > stuff' for 100M. Got it in memory first with untimed grep, then: $ time grep -i blah stuff bLAH blAH real 0m7.227s user 0m7.194s sys 0m0.011s $ time LANG=C grep -i blah stuff bLAH blAH real 0m0.486s user 0m0.467s sys 0m0.016s So a pretty big difference. My default lang is en_US.utf8.

UNIXgod 13y ago

It's historical and sets your local character set. The default value on FreeBSD is POSIX which is an alias the historically value which is C. Desktop unix's like OSX or ubuntu set it to utf8. Another note is that this is triggering fgrep which is already fast due to it's fixed length expression (i.e. no recursion is involved)