4 ms·

I'd like to share a recent experience on a related note, but in the opposite spirit - rather than reduce the number of command invocations on a command line, it

I'd like to share a recent experience on a related note, but in the opposite spirit - rather than reduce the number of command invocations on a command line, it may make sense to increase it. I had a loop operating on a text file like this: while read line do echo "$line" | sed -e "s/A/X/" -e "s/B/Y/" -e "s/C/Z/" ... Gradually, as I added more things to replace, I noticed severe slowdown. Things got fast again when I rewrote it as while read line do echo "$line" | sed -e "s/A/X/" | sed -e "s/B/Y/" | sed -e "s/C/Z/" ... Turns out the later (multiple processes in parallel) helped with throughput.

klodolph 9y ago
As a minor note, the first one can be written as "s/A/X;s/B/Y;s/C/Z/"
avaika 9y ago
why do you need `while read` part at all? this is the main slow down here. shell is extremely time consuming compared to sed/awk/whatever tool.
- aargh_aargh 9y ago
  The script is currently a filter - the input comes from stdin.
  - sebcat 9y ago
    You can redirect stdin to sed directly, e.g., cat << EOF | sed -e 's/A/X/;s/B/Y/;s/C/Z/;' A Brave Cow Was Walking Crows Were Airborne All Was Well EOF or, cat << EOF > myfile A Brave Cow Was Walking Crows Were Airborne All Was Well EOF sed -e 's/A/X/;s/B/Y/;s/C/Z/;' < myfile or (copy-pasted from shell for clarity), $ cat foo.sh #!/bin/sh sed -e 's/A/X/;s/B/Y/;s/C/Z/;' $ cat << EOF | ./foo.sh A Brave Cow EOF X Yrave Zow
    - aargh_aargh 9y ago
      Thank you, that helped. I had to get rid of '^' anchors (because it's not processing individual lines anymore) and I put sed -e '' < /dev/stdin into the script, but the solution is now much simpler and faster.
      - sebcat 9y ago
        sed works on stdin by default (see third example) so there shouldn't be a need to pipe from /dev/stdin, and line anchors should still work. $ cat myfile foo foo baz baz $ cat foo.sh #!/bin/sh sed 's/^foo/bar/;s/^baz/foobar/' $ ./foo.sh < myfile bar foo foobar baz Depending on your expression, you may want to use -E to get POSIX ERE syntax.

klodolph 9y ago

As a minor note, the first one can be written as "s/A/X;s/B/Y;s/C/Z/"

avaika 9y ago

why do you need `while read` part at all? this is the main slow down here. shell is extremely time consuming compared to sed/awk/whatever tool.

aargh_aargh 9y ago
The script is currently a filter - the input comes from stdin.
- sebcat 9y ago
  You can redirect stdin to sed directly, e.g., cat << EOF | sed -e 's/A/X/;s/B/Y/;s/C/Z/;' A Brave Cow Was Walking Crows Were Airborne All Was Well EOF or, cat << EOF > myfile A Brave Cow Was Walking Crows Were Airborne All Was Well EOF sed -e 's/A/X/;s/B/Y/;s/C/Z/;' < myfile or (copy-pasted from shell for clarity), $ cat foo.sh #!/bin/sh sed -e 's/A/X/;s/B/Y/;s/C/Z/;' $ cat << EOF | ./foo.sh A Brave Cow EOF X Yrave Zow
  - aargh_aargh 9y ago
    Thank you, that helped. I had to get rid of '^' anchors (because it's not processing individual lines anymore) and I put sed -e '' < /dev/stdin into the script, but the solution is now much simpler and faster.
    - sebcat 9y ago
      sed works on stdin by default (see third example) so there shouldn't be a need to pipe from /dev/stdin, and line anchors should still work. $ cat myfile foo foo baz baz $ cat foo.sh #!/bin/sh sed 's/^foo/bar/;s/^baz/foobar/' $ ./foo.sh < myfile bar foo foobar baz Depending on your expression, you may want to use -E to get POSIX ERE syntax.