4 ms·

>UNIX shells already optimise pipelines by running the commands along it concurrently. I am aware of that. Still, performing multiple operations on the data wi

>UNIX shells already optimise pipelines by running the commands along it concurrently. I am aware of that. Still, performing multiple operations on the data within a single loop and with less copying would be an optimization.

laumars 9y ago
I get what you're saying but technically those tools are running inside the same single loop (as that file is only iterated the once most of that pipeline) but distributed across multiple OS threads. Shell scripts are surprisingly efficient at pipelined work where you're just dealing with streaming data (as your example mostly does). It's the more complex logic they fall significantly short on. The weakest area and why I kept saying "most" of that pipeline would be the use of tail part way through the chain (as others have mentioned). But that could be optimised within Bash to something like: tail -r somefile | grep 'expression' | egrep 'expression' | head -n number | cut ... (I think -r flag on tail is GNU rather than POSIX though) You're object point is definitely relevant though. For example how many times have we seen shell scripts break because they don't consider spaces in filenames? Talking about annoyances in Bash I'd also throw in exception handling as a major problem for traditional shells. Having Bash fail in expected ways and handle them cleanly is a huge pain in the arse. Those above two points (and wanting cleaner syntax for iteration) are what drove me to write my own $SHELL so it does sound like I'm addressing some of your annoyances - albeit not all of them. However it's a long way from being something usable in production environments but I'm happy to access pull requests if you do ever feel you want to contribute to something that's trying to work towards at least some of your goals :)

4 ms·

>UNIX shells already optimise pipelines by running the commands along it concurrently. I am aware of that. Still, performing multiple operations on the data wi

by eriknstr 9y ago

laumars 9y ago
I get what you're saying but technically those tools are running inside the same single loop (as that file is only iterated the once most of that pipeline) but distributed across multiple OS threads. Shell scripts are surprisingly efficient at pipelined work where you're just dealing with streaming data (as your example mostly does). It's the more complex logic they fall significantly short on. The weakest area and why I kept saying "most" of that pipeline would be the use of tail part way through the chain (as others have mentioned). But that could be optimised within Bash to something like: tail -r somefile | grep 'expression' | egrep 'expression' | head -n number | cut ... (I think -r flag on tail is GNU rather than POSIX though) You're object point is definitely relevant though. For example how many times have we seen shell scripts break because they don't consider spaces in filenames? Talking about annoyances in Bash I'd also throw in exception handling as a major problem for traditional shells. Having Bash fail in expected ways and handle them cleanly is a huge pain in the arse. Those above two points (and wanting cleaner syntax for iteration) are what drove me to write my own $SHELL so it does sound like I'm addressing some of your annoyances - albeit not all of them. However it's a long way from being something usable in production environments but I'm happy to access pull requests if you do ever feel you want to contribute to something that's trying to work towards at least some of your goals :)