3 ms·

This feels great until your program reaches some complexity threshold and you start hitting core limitation of the shell execution model. Then you have to use h

by theamk 7y ago

This feels great until your program reaches some complexity threshold and you start hitting core limitation of the shell execution model. Then you have to use high-numbered file descriptors, named temporary files, PIPESTATUS, etc... - and your neat programming model becomes ugly and full of hacks. Sometimes shell is really the best even for complex stuff - for the example in initrd or during machine provisioning - but in most cases, having any sort of non-trivial flow control in shell is a bad idea.

enriquto 7y ago
> any sort of non-trivial flow control in shell is a bad idea. It is often a bad idea in other programming languages also. Yet, some complex control flow constructs are straigthforward in shell but very difficult in other languages. How do you ls *.jpg | grep 2019 | while read x ; do echo convert ... $x album/$x done | xargs -L 1 -P 32 in javascript or python, for example?
- jstimpfle 7y ago
  Wow, there are 7 ways this can easily fail, and the script might not even notice. How do you write in a normal language? With roughly 3 times the amount of keystrokes, but in a much more robust way. And if something fails, you will actually get a Python exception.
  - enriquto 7y ago
    If I understand correctly, the whole point of Oil is to make a shell in which this kind of code is perfectly correct and robust.
    - chubot 7y ago
      Yes exactly, see my sibling response. This is a perfect use case for the "xargs $0 dispatch" pattern I mentioned, and maybe I will use it in that blog post. Your example does have some quoting problems which is understandable / common, but in Oil the obvious way should be the right way. That's not true in shell! https://news.ycombinator.com/item?id=22161847 https://news.ycombinator.com/item?id=22161847 Thanks again for the encouragement!
- theamk 7y ago
  The only difficult part here is xargs parallel invocation. If you do this often, you may want to find/make the library for it. But even with just default python install, it is pretty straightforward: files = [f for f in glob("*.jpg") if "2019" in f] commands = ["convert ... {} album/{}".format(f) for f in files] subprocess.run("xargs -L 1 -P 32".split(), input=os.fsencode("\n".join(commands))) Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.
  - jstimpfle 7y ago
    In sh, the correct version would be for f in ./*2019*.jpg ; do convert "$f" album/"$f" # just don't use xargs done Note the prepended ./ which is necessary if you want to protect against filenames that look like command line switches (a file named "-rf" can be pretty dangerous) If you wanted to use xargs and still make it perfectly safe with regards to spaces and quoting, you could use a GNU extension: find . -maxdepth 1 -name '*2019*.jpg' -print0 | xargs -0 convertalbum.sh # convertalbum.sh convert "$1" album/"$1" find prints the files terminated by ASCII NUL characters, xargs can understand this format if you give it the -0 option. There are still problems around detecting errors in pipes, though.
    - enriquto 7y ago
      > just don't use xargs the whole point of my example was to show how easy parallelism is. You create a list of tasks, then you pipe the list to xargs or gnu parallel.
      - jstimpfle 7y ago
        You can't write ASCII NUL characters from portable shell, that's why this task cannot be done correctly (for all possible filenames) from shell. You need to use tools like find(1) So, no, it's not easy, at least not if you want to handle all cases without wreaking havoc.
        7y ago

by theamk 7y ago

enriquto 7y ago

> any sort of non-trivial flow control in shell is a bad idea. It is often a bad idea in other programming languages also. Yet, some complex control flow constructs are straigthforward in shell but very difficult in other languages. How do you ls *.jpg | grep 2019 | while read x ; do echo convert ... $x album/$x done | xargs -L 1 -P 32 in javascript or python, for example?

jstimpfle 7y ago
Wow, there are 7 ways this can easily fail, and the script might not even notice. How do you write in a normal language? With roughly 3 times the amount of keystrokes, but in a much more robust way. And if something fails, you will actually get a Python exception.
- enriquto 7y ago
  If I understand correctly, the whole point of Oil is to make a shell in which this kind of code is perfectly correct and robust.
  - chubot 7y ago
    Yes exactly, see my sibling response. This is a perfect use case for the "xargs $0 dispatch" pattern I mentioned, and maybe I will use it in that blog post. Your example does have some quoting problems which is understandable / common, but in Oil the obvious way should be the right way. That's not true in shell! https://news.ycombinator.com/item?id=22161847 https://news.ycombinator.com/item?id=22161847 Thanks again for the encouragement!
theamk 7y ago
The only difficult part here is xargs parallel invocation. If you do this often, you may want to find/make the library for it. But even with just default python install, it is pretty straightforward: files = [f for f in glob("*.jpg") if "2019" in f] commands = ["convert ... {} album/{}".format(f) for f in files] subprocess.run("xargs -L 1 -P 32".split(), input=os.fsencode("\n".join(commands))) Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.
- jstimpfle 7y ago
  In sh, the correct version would be for f in ./*2019*.jpg ; do convert "$f" album/"$f" # just don't use xargs done Note the prepended ./ which is necessary if you want to protect against filenames that look like command line switches (a file named "-rf" can be pretty dangerous) If you wanted to use xargs and still make it perfectly safe with regards to spaces and quoting, you could use a GNU extension: find . -maxdepth 1 -name '*2019*.jpg' -print0 | xargs -0 convertalbum.sh # convertalbum.sh convert "$1" album/"$1" find prints the files terminated by ASCII NUL characters, xargs can understand this format if you give it the -0 option. There are still problems around detecting errors in pipes, though.
  - enriquto 7y ago
    > just don't use xargs the whole point of my example was to show how easy parallelism is. You create a list of tasks, then you pipe the list to xargs or gnu parallel.
    - jstimpfle 7y ago
      You can't write ASCII NUL characters from portable shell, that's why this task cannot be done correctly (for all possible filenames) from shell. You need to use tools like find(1) So, no, it's not easy, at least not if you want to handle all cases without wreaking havoc.
      - 7y ago