4 ms·

All the OS X tools are horribly broken due to something fucked up inside their unicode support. try something like tr: time gtr a b < application.log > /dev

All the OS X tools are horribly broken due to something fucked up inside their unicode support. try something like tr: time gtr a b < application.log > /dev/null time tr a b < application.log > /dev/null if you set LC_ALL=c it might be a little faster. $ time gtr a b < http_10x.log > /dev/null real 0m0.061s user 0m0.032s sys 0m0.028s $ time tr a b < http_10x.log > /dev/null real 0m2.284s user 0m2.267s sys 0m0.014s

danieldk 8y ago
Yep. For this reason I override the standard UNIX tools on macOS with Nix + home-manager. E.g.: ~ % which grep /Users/daniel/.nix-profile/bin/grep ~ % which sed /Users/daniel/.nix-profile/bin/sed ~ % which ls /Users/daniel/.nix-profile/bin/ls
Twirrim 8y ago
So far as I can see, setting LC_ALL=c shaves about 5 seconds off the "foobarbaz" case, and nothing off the "^INFO" case.
scottlamb 8y ago
The same LC_ALL=C trick also speeds up GNU tools, sort in particular: $ shuf /var/log/syslog > shuf-syslog $ time sort shuf-syslog > /dev/null real 0m0.536s user 0m0.870s sys 0m0.031s $ time LC_ALL=C sort shuf-syslog > /dev/null real 0m0.094s user 0m0.109s sys 0m0.024s It's more dramatic with a bigger file, of course. I once got annoyed with sort's speed, threw together a parallel external sort program that dramatically outperformed it, then realized the difference was not as dramatic with LC_ALL=C...oh well, it was a fun afternoon program anyway.
- danudey 8y ago
  In fairness, when I'm dealing with unicode data which is actually unicode data, I don't want to set `LC_ALL=C` and fuck up my data set.
  - scottlamb 8y ago
    It's certainly worth considering, but the answer depends on what you're doing with it. If your goal is to binary search over it for an exact string match, for example, the important thing is that your sorting and searching code are consistent. Might as well have both use the simpler/faster LC_ALL=C semantics.
  - treffer 8y ago
    Actually sorting should be fine. The order of UTF-8 will always be the same as the codepoints encoded under it if you sort it as binary. I really appreciate this property of UTF-8, and I can highly recommend to do a pen&paper exercise to see why it preserves order :-) It wouldn't work if you want any type of normalisation and especially solving composition/decomposition.
    - burntsushi 8y ago
      Locale specific sorting is not the same as sorting by codepoint. The Unicode collation algorithm is fairly involved: https://www.unicode.org/reports/tr10/ https://www.unicode.org/reports/tr10/
- james_s_tayler 8y ago
  If you are just using sort because you need to use uniq then you can replace the whole thing with an awk command that gets unique lines without sorting. I think it's faster.

danieldk 8y ago

Yep. For this reason I override the standard UNIX tools on macOS with Nix + home-manager. E.g.: ~ % which grep /Users/daniel/.nix-profile/bin/grep ~ % which sed /Users/daniel/.nix-profile/bin/sed ~ % which ls /Users/daniel/.nix-profile/bin/ls

Twirrim 8y ago

So far as I can see, setting LC_ALL=c shaves about 5 seconds off the "foobarbaz" case, and nothing off the "^INFO" case.

scottlamb 8y ago

The same LC_ALL=C trick also speeds up GNU tools, sort in particular: $ shuf /var/log/syslog > shuf-syslog $ time sort shuf-syslog > /dev/null real 0m0.536s user 0m0.870s sys 0m0.031s $ time LC_ALL=C sort shuf-syslog > /dev/null real 0m0.094s user 0m0.109s sys 0m0.024s It's more dramatic with a bigger file, of course. I once got annoyed with sort's speed, threw together a parallel external sort program that dramatically outperformed it, then realized the difference was not as dramatic with LC_ALL=C...oh well, it was a fun afternoon program anyway.

danudey 8y ago
In fairness, when I'm dealing with unicode data which is actually unicode data, I don't want to set `LC_ALL=C` and fuck up my data set.
- scottlamb 8y ago
  It's certainly worth considering, but the answer depends on what you're doing with it. If your goal is to binary search over it for an exact string match, for example, the important thing is that your sorting and searching code are consistent. Might as well have both use the simpler/faster LC_ALL=C semantics.
- treffer 8y ago
  Actually sorting should be fine. The order of UTF-8 will always be the same as the codepoints encoded under it if you sort it as binary. I really appreciate this property of UTF-8, and I can highly recommend to do a pen&paper exercise to see why it preserves order :-) It wouldn't work if you want any type of normalisation and especially solving composition/decomposition.
  - burntsushi 8y ago
    Locale specific sorting is not the same as sorting by codepoint. The Unicode collation algorithm is fairly involved: https://www.unicode.org/reports/tr10/ https://www.unicode.org/reports/tr10/
james_s_tayler 8y ago
If you are just using sort because you need to use uniq then you can replace the whole thing with an awk command that gets unique lines without sorting. I think it's faster.