4 ms·
> any sort of non-trivial flow control in shell is a bad idea. It is often a bad idea in other programming languages also. Yet, some complex control flow cons
by enriquto 7y ago
> any sort of non-trivial flow control in shell is a bad idea.
It is often a bad idea in other programming languages also. Yet, some complex control flow constructs are straigthforward in shell but very difficult in other languages. How do you
ls *.jpg | grep 2019 | while read x ; do
echo convert ... $x album/$x
done | xargs -L 1 -P 32
in javascript or python, for example?
- jstimpfle 7y agoWow, there are 7 ways this can easily fail, and the script might not even notice. How do you write in a normal language? With roughly 3 times the amount of keystrokes, but in a much more robust way. And if something fails, you will actually get a Python exception.
- enriquto 7y agoIf I understand correctly, the whole point of Oil is to make a shell in which this kind of code is perfectly correct and robust.
- chubot 7y agoYes exactly, see my sibling response. This is a perfect use case for the "xargs $0 dispatch" pattern I mentioned, and maybe I will use it in that blog post. Your example does have some quoting problems which is understandable / common, but in Oil the obvious way should be the right way. That's not true in shell! https://news.ycombinator.com/item?id=22161847 https://news.ycombinator.com/item?id=22161847 Thanks again for the encouragement!
- theamk 7y agoThe only difficult part here is xargs parallel invocation. If you do this often, you may want to find/make the library for it. But even with just default python install, it is pretty straightforward: files = [f for f in glob("*.jpg") if "2019" in f] commands = ["convert ... {} album/{}".format(f) for f in files] subprocess.run("xargs -L 1 -P 32".split(), input=os.fsencode("\n".join(commands))) Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.
- jstimpfle 7y agoIn sh, the correct version would be for f in ./*2019*.jpg ; do convert "$f" album/"$f" # just don't use xargs done Note the prepended ./ which is necessary if you want to protect against filenames that look like command line switches (a file named "-rf" can be pretty dangerous) If you wanted to use xargs and still make it perfectly safe with regards to spaces and quoting, you could use a GNU extension: find . -maxdepth 1 -name '*2019*.jpg' -print0 | xargs -0 convertalbum.sh # convertalbum.sh convert "$1" album/"$1" find prints the files terminated by ASCII NUL characters, xargs can understand this format if you give it the -0 option. There are still problems around detecting errors in pipes, though.
- enriquto 7y ago> just don't use xargs the whole point of my example was to show how easy parallelism is. You create a list of tasks, then you pipe the list to xargs or gnu parallel.
- jstimpfle 7y agoYou can't write ASCII NUL characters from portable shell, that's why this task cannot be done correctly (for all possible filenames) from shell. You need to use tools like find(1) So, no, it's not easy, at least not if you want to handle all cases without wreaking havoc.
- chubot 7y agoPractically speaking, you can write NUL from shell because bash is everywhere, busybox ash emulates a lot of bash, etc. However I won't claim it's pretty or easy for people to use, so the point of Oil is basically to clean up patterns like this, e.g. find -print0 | xargs -0 and more. This post uses a related problem and solution as design motivation for Oil: How to Quickly and Correctly Generate a Git Log in HTML http://www.oilshell.org/blog/2017/09/19.html http://www.oilshell.org/blog/2017/09/19.html
- enriquto 7y ago