4 ms·

@chubot: There is one use-case I come across every once in a while: https://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-

by JepZ 9y ago

@chubot: There is one use-case I come across every once in a while: https://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-finish-and-return-exit-code-0 https://stackoverflow.com/questions/356100/how-to-wait-in-ba... So whenever you want to do things in parallel there is probably a limit to the number of processes you would like to execute in parallel (e.g. the famous compiler limit formula: number of CPU cores +1). It would be great if Oil could support such a use-case out of the box, as easy parallelism without the ability to artificially limit the number of parallel executions is often useless.

chubot 9y ago
Absolutely. In fact, the bash manual explicitly refers to GNU parallel for this use case! I use xargs -P all over the Oil codebase, which does what you want. The trick is to end the file with "$@", and then invoke xargs -P 4 -- $0 my-func. That way xargs can run arbitrary shell functions, not just something like sh -c "..." ! I'm going to write a blog post about this. I also do this with find -exec $0 myfunc ';' https://github.com/oilshell/oil/blob/master/test/spec-runner.sh#L220 https://github.com/oilshell/oil/blob/master/test/spec-runner... However I think Oil will have something built-in to make this parallel process more friendly. I will probably implement xargs so it can run my own shell scripts without GNU xargs, but then add a nicer syntax. (Probably "each", since that's what xargs really does.) This gets into your other question about standard utils, which I'll answer now. Short answer: yes I would like something like that, it's just a matter of development time and priorities. I agree with the problem you point out.
- JepZ 9y ago
  Sounds pretty cool. My biggest problems with xargs is that I had constantly some weird edge cases, so I try to avoid it. As GNU parallel doesn't seem to be part of standard installations it would be an external dependency for a script, which I try to avoid too. So I ended up using the loop syntax: for i in {0..9}; do echo "$i" & done wait It is not so Unix like, but I find it easier to debug. It would be great if Oil would have a solution for limiting that kind of parallel execution too. I am aware that this isn't simple as there are different options here how to implement it (global limit vs. local limit vs. named limit). Just an idea from the top of my head an idea for an optional named limit: Lets call it 'flow': flow [options] [command]': -n number of max parallel processes -c (optional) identifier of the counter. Example: for i in {0..9}; do flow -c myCounter -n 4 echo "$i" done Just an idea.

by JepZ 9y ago

chubot 9y ago

Absolutely. In fact, the bash manual explicitly refers to GNU parallel for this use case! I use xargs -P all over the Oil codebase, which does what you want. The trick is to end the file with "$@", and then invoke xargs -P 4 -- $0 my-func. That way xargs can run arbitrary shell functions, not just something like sh -c "..." ! I'm going to write a blog post about this. I also do this with find -exec $0 myfunc ';' https://github.com/oilshell/oil/blob/master/test/spec-runner.sh#L220 https://github.com/oilshell/oil/blob/master/test/spec-runner... However I think Oil will have something built-in to make this parallel process more friendly. I will probably implement xargs so it can run my own shell scripts without GNU xargs, but then add a nicer syntax. (Probably "each", since that's what xargs really does.) This gets into your other question about standard utils, which I'll answer now. Short answer: yes I would like something like that, it's just a matter of development time and priorities. I agree with the problem you point out.

JepZ 9y ago
Sounds pretty cool. My biggest problems with xargs is that I had constantly some weird edge cases, so I try to avoid it. As GNU parallel doesn't seem to be part of standard installations it would be an external dependency for a script, which I try to avoid too. So I ended up using the loop syntax: for i in {0..9}; do echo "$i" & done wait It is not so Unix like, but I find it easier to debug. It would be great if Oil would have a solution for limiting that kind of parallel execution too. I am aware that this isn't simple as there are different options here how to implement it (global limit vs. local limit vs. named limit). Just an idea from the top of my head an idea for an optional named limit: Lets call it 'flow': flow [options] [command]': -n number of max parallel processes -c (optional) identifier of the counter. Example: for i in {0..9}; do flow -c myCounter -n 4 echo "$i" done Just an idea.