3 ms·

Show HN: Xzgrep is twice as fast with dash versus bash

2 points by chasil 4y ago

In the CentOS world, a shell script written by "Charles Levert" has been extended as the compressed grep for (gzip/bzip2/xz) as /bin/(zgrep/bzgrep/xzgrep); variants of this script are bundled in their respective packages. The dash shell, available in EPEL for CentOS, appears to specifically speed up xzgrep considerably when altered from #!/bin/sh to #!/bin/dash. In my testing on a 32-bit platform under xargs, a search of 14k files of 20M total size runs in half the time. Do others see this performance increase? Is this another tangible benefit for Debian/Ubuntu moving the system shell from bash to their Almquist shell derivative?

1vuio0pswjnm7 4y ago
If the script was using bash, a noticeable speedup when using dash instead is not surprising. Having used NetBSD's sh for many years as an interactive shell, I find bash on Linux is far too bloated and slow. I modified dash (a derivative of NetBSD sh) for use as an interactive shell as well as a non-interactive one by adding tabcompletion. I like to write quick scripts without having to include #!/bin/sh at the top. I get no benefit from having two shells: one interactive and the other non-interactive.
stop50 4y ago
Dash was developed to be more posix compatible.
- chasil 4y ago
  The Almquist shell first appeared in 1989, which predates the POSIX.2 standard shell from 1992. The new POSIX features were likely retrofitted onto the code. https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/POSIX#POSIX.2 https://en.wikipedia.org/wiki/POSIX#POSIX.2 Bash was introduced in the same year as the Almquist shell. https://en.wikipedia.org/wiki/Bash_(Unix_shell) https://en.wikipedia.org/wiki/Bash_(Unix_shell)
LinuxBender 4y ago
Have you traced what system calls xzgrep is making in one shell vs the other? I ask because there are a few odd behaviors in each shell that can slow down some operations. A funny one I ran into was not setting the TZ variable and thus slowing down some system calls. This was also on CentOS which I have not used in a while.
- chasil 4y ago
  I have not done an strace. I did try a much larger dataset, and the speed advantage does decrease. Test files - number and total size: $ ls *.xz | wc -l 59908 $ du -b *.xz | awk '{s += $1}; END {print s}' 444456636 $ echo $((444456636 /1024 /1024)) 423 Dash trial: # head -1 /usr/bin/xzgrep #!/bin/dash # time xzgrep 1234567 *.xz 2> /dev/null ... real 16m41.275s user 8m50.386s sys 11m0.066s # time xzgrep 1234567 *.xz 2> /dev/null ... real 16m1.004s user 8m41.904s sys 10m45.139s Bash trial: # vi /usr/bin/xzgrep # head -1 /usr/bin/xzgrep #!/bin/sh # time xzgrep 1234567 *.xz 2> /dev/null ... real 21m53.975s user 7m23.522s sys 22m10.284s p.s. The wrapper script fails for filenames that have spaces. It could definitely use some developer attention.

2 points by chasil 4y ago

1vuio0pswjnm7 4y ago

If the script was using bash, a noticeable speedup when using dash instead is not surprising. Having used NetBSD's sh for many years as an interactive shell, I find bash on Linux is far too bloated and slow. I modified dash (a derivative of NetBSD sh) for use as an interactive shell as well as a non-interactive one by adding tabcompletion. I like to write quick scripts without having to include #!/bin/sh at the top. I get no benefit from having two shells: one interactive and the other non-interactive.

stop50 4y ago

Dash was developed to be more posix compatible.

chasil 4y ago
The Almquist shell first appeared in 1989, which predates the POSIX.2 standard shell from 1992. The new POSIX features were likely retrofitted onto the code. https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/Almquist_shell https://en.wikipedia.org/wiki/POSIX#POSIX.2 https://en.wikipedia.org/wiki/POSIX#POSIX.2 Bash was introduced in the same year as the Almquist shell. https://en.wikipedia.org/wiki/Bash_(Unix_shell) https://en.wikipedia.org/wiki/Bash_(Unix_shell)

LinuxBender 4y ago

Have you traced what system calls xzgrep is making in one shell vs the other? I ask because there are a few odd behaviors in each shell that can slow down some operations. A funny one I ran into was not setting the TZ variable and thus slowing down some system calls. This was also on CentOS which I have not used in a while.

chasil 4y ago
I have not done an strace. I did try a much larger dataset, and the speed advantage does decrease. Test files - number and total size: $ ls *.xz | wc -l 59908 $ du -b *.xz | awk '{s += $1}; END {print s}' 444456636 $ echo $((444456636 /1024 /1024)) 423 Dash trial: # head -1 /usr/bin/xzgrep #!/bin/dash # time xzgrep 1234567 *.xz 2> /dev/null ... real 16m41.275s user 8m50.386s sys 11m0.066s # time xzgrep 1234567 *.xz 2> /dev/null ... real 16m1.004s user 8m41.904s sys 10m45.139s Bash trial: # vi /usr/bin/xzgrep # head -1 /usr/bin/xzgrep #!/bin/sh # time xzgrep 1234567 *.xz 2> /dev/null ... real 21m53.975s user 7m23.522s sys 22m10.284s p.s. The wrapper script fails for filenames that have spaces. It could definitely use some developer attention.