4 ms·

I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Py

I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Python that's generally a better idea than GNU Parallel, IMO.

globular-toast 5y ago
The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.
- adwn 5y ago
  > that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.
  - globular-toast 5y ago
    Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.
    - adwn 5y ago
      > I'm talking about the processes that can be split up in that way No, you weren't. You said: "[...] GNU parallel [...] works for any non-parallel process" (emphasis mine) > Those problems are so common that someone literally wrote GNU parallel to solve them. As part of my job I'm writing multi-threaded, parallel programs all the time, and in those years only a single problem would have been feasible to parallelize with GNU parallel; but since I was using Rust, it was trivial to do the parallelization right there in my code without having to resort an outer script/binary that calls GNU parallel on my program.
  - MainJane 5y ago
    > Try using GNU parallel to sort a list of numbers, `parsort` is part of GNU Parallel.
    - adwn 5y ago
      ... and it uses a manually implemented post-processing step. You can't just run the sort program with GNU parallel and expect to get a fully sorted list.
      - MainJane 5y ago
        > Try using GNU parallel to sort a list of numbers, [...] – it's not possible, Yet it clearly is possible, so your blanket statement is clearly wrong. `parsort` a simple wrapper, and this really goes for many uses of GNU Parallel: You need to prepare your data for the parallel step and post-process the output. Maybe you originally meant to say: "Only for those processes, where you can preprocess the input and post-process the outputs."
        adwn 5y ago
        Why would you use GNU parallel if you have to implement your own non-trivial pre- or post-processing logic anyway? Just spawn the worker processes yourself. GNU parallel is great if you have, e.g., a bunch of files, each of which needs to be processed individually, like running awk or sed over it. Then you can just plop parallel in front and get a speedup for free. That's not what parsort does.

globular-toast 5y ago

The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.

adwn 5y ago
> that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.
- globular-toast 5y ago
  Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.
  - adwn 5y ago
    > I'm talking about the processes that can be split up in that way No, you weren't. You said: "[...] GNU parallel [...] works for any non-parallel process" (emphasis mine) > Those problems are so common that someone literally wrote GNU parallel to solve them. As part of my job I'm writing multi-threaded, parallel programs all the time, and in those years only a single problem would have been feasible to parallelize with GNU parallel; but since I was using Rust, it was trivial to do the parallelization right there in my code without having to resort an outer script/binary that calls GNU parallel on my program.
- MainJane 5y ago
  > Try using GNU parallel to sort a list of numbers, `parsort` is part of GNU Parallel.
  - adwn 5y ago
    ... and it uses a manually implemented post-processing step. You can't just run the sort program with GNU parallel and expect to get a fully sorted list.
    - MainJane 5y ago
      > Try using GNU parallel to sort a list of numbers, [...] – it's not possible, Yet it clearly is possible, so your blanket statement is clearly wrong. `parsort` a simple wrapper, and this really goes for many uses of GNU Parallel: You need to prepare your data for the parallel step and post-process the output. Maybe you originally meant to say: "Only for those processes, where you can preprocess the input and post-process the outputs."
      - adwn 5y ago
        Why would you use GNU parallel if you have to implement your own non-trivial pre- or post-processing logic anyway? Just spawn the worker processes yourself. GNU parallel is great if you have, e.g., a bunch of files, each of which needs to be processed individually, like running awk or sed over it. Then you can just plop parallel in front and get a speedup for free. That's not what parsort does.