3 ms·

Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.

by kamaal 7y ago

masklinn 7y ago
Don't see why it "doesn't scale well", the overhead is roughly constant: use the stdout of one program as the input of the next.
- kamaal 7y ago
  That's not as easy as cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' You end up writing too much code, it's very verbose. Some times symbols are what you want. In fact the biggest progress in the growth of Math happened when they tossed out doing math with words and bought in symbols.
  - Too 7y ago
    You forgot setting -e and -o pipefail. In case "something" is missing or can't be read for some other reason, your script will just continue happily without warning. But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable. Not so simple after all.
  - masklinn 7y ago
    > That's not as easy as > cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' Literally none of this is actually useful if you're already in Python: ( line.split('\t')[0].replace(…, …) for line in open('something') if 'this' in line ) > You end up writing too much code I can believe that if you're calling to external processes to perform operations which are pretty much trivial in the language.
    - kamaal 7y ago
      Of course, if you plan to use one $language alone, you can do anything in that $language. This question is specific to pipes.
      - masklinn 7y ago
        > Of course, if you plan to use one $language alone, you can do anything in that $language. That's missing what I'm noting though, which is that you don't need pipes anywhere near a shell script if you can simply do more of your work in-language. Case in point being that the entire pipeline you cited as an issue has no reason to exist outside of a shell or shell script.
        kamaal 7y ago
        That's because regexes are an another pain in Python, You have to write a lot of boiler plate exception handling code. Plus the same boiler plate code for file operations. It doesn't feel like the language was designed for these tasks.
laurieodgers 7y ago
I found this problem as well when doing a direct port including pipes from bash to python. It was the first iteration into python and I was lacking time to do a proper port into python native. The python code to do piping ends up as longwinded and the plumbing of pipes ends up a massive headache so I wrote tidycmd to overcome that issue https://github.com/laurieodgers/tidycmd https://github.com/laurieodgers/tidycmd

Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.

by kamaal 7y ago

Doesn't scale well when you have to stitch more than two commands with pipes, which is a very common use case for Unix CLI utilities.

masklinn 7y ago

Don't see why it "doesn't scale well", the overhead is roughly constant: use the stdout of one program as the input of the next.

kamaal 7y ago
That's not as easy as cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' You end up writing too much code, it's very verbose. Some times symbols are what you want. In fact the biggest progress in the growth of Math happened when they tossed out doing math with words and bought in symbols.
- Too 7y ago
  You forgot setting -e and -o pipefail. In case "something" is missing or can't be read for some other reason, your script will just continue happily without warning. But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable. Not so simple after all.
- masklinn 7y ago
  > That's not as easy as > cat something | grep "this" | cut -f 1 | sed -e 's/.../.../' Literally none of this is actually useful if you're already in Python: ( line.split('\t')[0].replace(…, …) for line in open('something') if 'this' in line ) > You end up writing too much code I can believe that if you're calling to external processes to perform operations which are pretty much trivial in the language.
  - kamaal 7y ago
    Of course, if you plan to use one $language alone, you can do anything in that $language. This question is specific to pipes.
    - masklinn 7y ago
      > Of course, if you plan to use one $language alone, you can do anything in that $language. That's missing what I'm noting though, which is that you don't need pipes anywhere near a shell script if you can simply do more of your work in-language. Case in point being that the entire pipeline you cited as an issue has no reason to exist outside of a shell or shell script.
      - kamaal 7y ago
        That's because regexes are an another pain in Python, You have to write a lot of boiler plate exception handling code. Plus the same boiler plate code for file operations. It doesn't feel like the language was designed for these tasks.

laurieodgers 7y ago

I found this problem as well when doing a direct port including pipes from bash to python. It was the first iteration into python and I was lacking time to do a proper port into python native. The python code to do piping ends up as longwinded and the plumbing of pipes ends up a massive headache so I wrote tidycmd to overcome that issue https://github.com/laurieodgers/tidycmd https://github.com/laurieodgers/tidycmd