5 ms·

This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. The PoSix locale

by luriel 15y ago

This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken. I wasted days tracking down a bug when we moved a script from FreeBSD to a linux sytem that had an UTF-8 locale, even for plain ASCII input GNU awk could match lines erratically if the locale was not set back to C! I'm sure this bug has been fixed by now, but is by far not the only time I have found serious flaws with UTF-8 support in GNU tools, it seems this days the only way to be safe is to set the locale to C at the top of all your scripts, which is quite sad. This is also required because due to POSIX locale insanity the actual behavior of tools as basic as ls(1), and more fundamentally regexps matching changes based on the locale, making scripts that don't set their locale at the top unreliable and nonportable. Or using a toolkit with decent UTF-8 support, like the from Plan 9 that one can use on *nix systems via Plan 9 from User Space ( http://plan9.us http://plan9.us ) or 9base ( http://tools.suckless.org/9base http://tools.suckless.org/9base ), if you use those also your scripts will be much more portable, as there are all kinds of incompatibilities among for example awk implementations.

4ad 15y ago
Adding to what you said, I also find Plan9 regular expressions to be significantly easier to grok in your head compared to POSIX or PCRE regular expressions that push a lot of complexity on the user's side (PCREs are not even true regexps as PCREs are not regular). Plan9 regexps are very easy to reason about, more explicit and don't look as line noise as much.
- omaranto 15y ago
  I thought POSIX regexes allowed back references (as in "(*.)\1"), so they aren't regular either.
- adobriyan 15y ago
  RE with backreferences is strictly more powerful then RE without them. Implying that PCRE is somehow inferior due to backreferences support is quite fuddy.
dododo 15y ago
here's another "fun" grep locale oddity: $ echo HI | LANG=en_US.utf8 grep '^[a-z]' HI $ echo HI | LANG=C grep '^[a-z]' $ apparently en_{GB,US}.utf8 orders a-z like aAbBcC..zZ. $ echo ZI | LANG=en_US.utf8 grep '^[a-z]' $
- iso8859-1 15y ago
  This is what I get: $ echo HI | LANG=C grep '^[a-z]' $ echo HI | LANG=en_US.utf8 grep '^[a-z]' $ How come?
  - p9idf 15y ago
    I was able to reproduce the bug. It could be a version thing. ; grep --version GNU grep 2.6.3 ; echo A | LANG=en_US.utf8 grep '[a-z]' A
    - Ives 15y ago
      No, I have the same version but not a similar result. I also have the en_US.utf8 locale installed.
- ot 15y ago
  I had the same problem with sort: $ sort <<EOF > Aa > aa > Ab > ab > EOF aa Aa ab Ab I was going crazy because I was getting different results in OSX and Ubuntu. Setting the LANG to POSIX fixed it.
  - Macha 15y ago
    $ sort --version sort (GNU coreutils) 8.14 For what it's worth, this gets me the same results under LANG=C and LANG=en_US.utf8
EdiX 15y ago
> This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. > > The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken. Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.
- 4ad 15y ago
  The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that > dealing with Unicode correctly is a very expensive business is plain FUD. UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.
  - EdiX 15y ago
    > The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that >> dealing with Unicode correctly is a very expensive business > is plain FUD. If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.
    - morsch 15y ago
      > If you think this is true you don't know unicode enough. Yes, well, that is sort of a given for any statement about unicode (including this one).
  - 15y ago
pdw 15y ago
FWIW, more recent GNU grep releases (2.6 and later) run fine in UTF-8 mode. Actually, why is SmartOS, "the Completely Modern Operating System", shipping with a five year old grep?
- wollw 15y ago
  When the author shows the output of "which grep" it shows the path "/opt/local/bin/grep"; "/opt" is typically for additional installations of software. I haven't used SmartOS myself but my best guess is someone installed GNU grep manually and the PATH just included "/opt/local/bin" before "/usr/bin".
  - pdw 15y ago
    I guess SmartOS reserves /usr for the Solaris base install and puts additions in /opt/local. If you look at http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Datasets http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Dat... and click "to see list of installed software", it includes grep 2.5.3.
- rg3 15y ago
  Minor correction: the "bug" in grep was fixed in GNU grep 2.7, not 2.6. I recall that because I wrote an article about it on my blog. See the release notes for GNU grep 2.7 here: http://savannah.gnu.org/forum/forum.php?forum_id=6521 http://savannah.gnu.org/forum/forum.php?forum_id=6521
iradik 15y ago
Yup, the author an older version of grep 2.5.3. I have grep 2.6.3 on RHEL6 and only got a 2x boost doing a similar test.
iradik 15y ago
OP needs to upgrade grep to the latest version. He is using 2.5.3, and latest version is 2.10. Release notes for 2.6: "This release fixes an unexpectedly large number of flaws, from outright bugs (surprisingly many, considering this is "grep") to some occasionally debilitating performance problems." http://savannah.gnu.org/forum/forum.php?forum_id=6249 http://savannah.gnu.org/forum/forum.php?forum_id=6249

by luriel 15y ago

4ad 15y ago

Adding to what you said, I also find Plan9 regular expressions to be significantly easier to grok in your head compared to POSIX or PCRE regular expressions that push a lot of complexity on the user's side (PCREs are not even true regexps as PCREs are not regular). Plan9 regexps are very easy to reason about, more explicit and don't look as line noise as much.

omaranto 15y ago
I thought POSIX regexes allowed back references (as in "(*.)\1"), so they aren't regular either.
adobriyan 15y ago
RE with backreferences is strictly more powerful then RE without them. Implying that PCRE is somehow inferior due to backreferences support is quite fuddy.

dododo 15y ago

here's another "fun" grep locale oddity: $ echo HI | LANG=en_US.utf8 grep '^[a-z]' HI $ echo HI | LANG=C grep '^[a-z]' $ apparently en_{GB,US}.utf8 orders a-z like aAbBcC..zZ. $ echo ZI | LANG=en_US.utf8 grep '^[a-z]' $

iso8859-1 15y ago
This is what I get: $ echo HI | LANG=C grep '^[a-z]' $ echo HI | LANG=en_US.utf8 grep '^[a-z]' $ How come?
- p9idf 15y ago
  I was able to reproduce the bug. It could be a version thing. ; grep --version GNU grep 2.6.3 ; echo A | LANG=en_US.utf8 grep '[a-z]' A
  - Ives 15y ago
    No, I have the same version but not a similar result. I also have the en_US.utf8 locale installed.
ot 15y ago
I had the same problem with sort: $ sort <<EOF > Aa > aa > Ab > ab > EOF aa Aa ab Ab I was going crazy because I was getting different results in OSX and Ubuntu. Setting the LANG to POSIX fixed it.
- Macha 15y ago
  $ sort --version sort (GNU coreutils) 8.14 For what it's worth, this gets me the same results under LANG=C and LANG=en_US.utf8

EdiX 15y ago

> This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. > > The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken. Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used: http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu... The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.

4ad 15y ago
The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that > dealing with Unicode correctly is a very expensive business is plain FUD. UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.
- EdiX 15y ago
  > The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that >> dealing with Unicode correctly is a very expensive business > is plain FUD. If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.
  - morsch 15y ago
    > If you think this is true you don't know unicode enough. Yes, well, that is sort of a given for any statement about unicode (including this one).
- 15y ago

pdw 15y ago

FWIW, more recent GNU grep releases (2.6 and later) run fine in UTF-8 mode. Actually, why is SmartOS, "the Completely Modern Operating System", shipping with a five year old grep?

wollw 15y ago
When the author shows the output of "which grep" it shows the path "/opt/local/bin/grep"; "/opt" is typically for additional installations of software. I haven't used SmartOS myself but my best guess is someone installed GNU grep manually and the PATH just included "/opt/local/bin" before "/usr/bin".
- pdw 15y ago
  I guess SmartOS reserves /usr for the Solaris base install and puts additions in /opt/local. If you look at http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Datasets http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Dat... and click "to see list of installed software", it includes grep 2.5.3.
rg3 15y ago
Minor correction: the "bug" in grep was fixed in GNU grep 2.7, not 2.6. I recall that because I wrote an article about it on my blog. See the release notes for GNU grep 2.7 here: http://savannah.gnu.org/forum/forum.php?forum_id=6521 http://savannah.gnu.org/forum/forum.php?forum_id=6521

iradik 15y ago

Yup, the author an older version of grep 2.5.3. I have grep 2.6.3 on RHEL6 and only got a 2x boost doing a similar test.

iradik 15y ago

OP needs to upgrade grep to the latest version. He is using 2.5.3, and latest version is 2.10. Release notes for 2.6: "This release fixes an unexpectedly large number of flaws, from outright bugs (surprisingly many, considering this is "grep") to some occasionally debilitating performance problems." http://savannah.gnu.org/forum/forum.php?forum_id=6249 http://savannah.gnu.org/forum/forum.php?forum_id=6249