6 ms·

I'm sure that with tools like MPI-Bash [1] and more generally libcircle [2] many embarrassingly parallelizable problems can easily be tackled with standard *nix

by tobias2014 6y ago

I'm sure that with tools like MPI-Bash [1] and more generally libcircle [2] many embarrassingly parallelizable problems can easily be tackled with standard *nix tools. [1] https://github.com/lanl/MPI-Bash https://github.com/lanl/MPI-Bash [2] http://hpc.github.io/libcircle/ http://hpc.github.io/libcircle/

pdimitar 6y ago
I have lately used a shell `for` loop that only emits part of the iterated values (not all are eligible for further processing) and then fed the loop directly to the `parallel` tool via the pipe operator. The results were indeed embarrassingly parallel. I am a fan of some languages that seem better equipped to utilise our modern many-core machines and I'd still write a longer-living system with better guarantees in those languages -- but many people ignore shell goodness at their own peril.
- mturmon 6y ago
  Big agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads. I've been developing a simulation software that does Monte Carlo over realizations of a simulated universe, and xargs and (later) parallel were super useful for parts of the workload. All the parallel job instances run the same simulation code, but for a different simulated universe, all controlled by a random number seed, so you can generate an ensemble of simulations with basically: head -NumberOfSimsWanted seeds.txt | xargs simulation.py
  - pdimitar 6y ago
    Yep! Even as a fan of more parallel-friendly programming languages I still think having single-core algorithms applied to different input parameters in parallel is a very viable alternative approach if your problem space allows for it. I quite disliked the journey. The UNIX shell legacy shenanigans are real and can be a pain. But you can still bear a lot of fruit without going to the more arcane corners of it.
  - flyinghamster 6y ago
    I wound up writing a hybrid bash script with embedded Makefile to re-encode my FLAC collection to Opus. It was nice to be able to use -j16 on that task. Switching to Opus saved a great deal of space on my phone's SD card.
  - eythian 6y ago
    I have lately been working on a genetic algorithm system, written in single-threaded Perl. When I started off wanting to do, say, 20 simulations, I just used parallel, and it worked great on the 8 threads of my desktop machine. Now I've done a bit more and want to do long-running simulations that might take a few hours, I've taken to just firing it all at AWS Batch and having a script that uploads the results to S3. Now I can do 1-200 instances at the same time, which my own hardware isn't up to in a reasonable timeframe.
    - pdimitar 6y ago
      I have a pretty strong workstation (iMac Pro) that I currently don't use all the time. And a gaming PC that I barely touch in the last 9-10 months. Hit me up if you want me to lend you some CPU power sometime.
      - eythian 6y ago
        Cheers, but the configuring of it would be more hassle than it's worth I suspect. As it is, using AWS is very cheap as I can have it use spot instances. So hundreds of CPU-hours used for in the order of US$10-20.
        pdimitar 6y ago
        I suspect setting that up would indeed be a hassle. :) Thanks for mentioning AWS Batch, I didn't know about it and will look it up.
- qchris 6y ago
  I've done something very similar recently, for processing hundreds of GB of .csv files across thousands of records. I ended up writing a fast single-threaded Rust program for the file parsing, then fed it into a bash script running GNU Parallel. That thing flies, and watching the system monitor showing every thread on the system get pegged to 100% simultaneously is a neat experience. I'm sure I could have done the parallelization itself in Rust given enough time, but honestly I found the `parallel` command to be so easy and resulting in so little overhead that it didn't really even seem worth it to spend the time on a language-native solution.
  - spockz 6y ago
    Did the 100% system utilisation also lead to better performance? More cpu utilisation doesn’t always mean faster or higher throughput.
    - pdimitar 6y ago
      This can't be generalised but in many cases it did help me for CPU-bound tasks. Obviously there are cases where parallelisation is very superfluous. F.ex. if you have 1000 records of something, splitting the work of processing those (and we're talking really light processing like JSON encoding) among 4-8 threads is actually detrimental to performance.
    - qchris 6y ago
      That's certainly true, but it does look like in this case. I did the preliminary benchmarking on an 8-thread CPU (the actual HPC cluster has higher counts), and the total execution time on the processing was just under 8x, which I imagine came from the OS-level thread management overhead. I still need to check that pattern holds once thread counts increases, but I'm fairly optimistic it'll work out as long as I don't run into thermal throttling or I/O limits.
  - heresie-dabord 6y ago
    GNU Parallel with awk is a SAPANS, effective combination for data analysis. No faddish rube-goldberg machines. https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/ *SAPANS: as simple as possible, and no simpler Do note that awk and GNU Parallel are tools that use/contain C-like syntax, which is abhorrent to some factions. GNU Parallel is written in Perl.
- wott 6y ago
  At lot of people do not like to use the features already provided by their environment and like to take a single element of the original features set and reimplement the same features within that single element. Sometimes it improves performance, sometimes not. Sometimes it improves productivity, sometimes not. I am not sure of the global balance. * The OS provides process isolation and management, as well as various IPCs to deal with their interconnection? Let's use a single process and create threads, and then fibres. More performance? yes, a bit; more headaches? yes, many. * The display protocol/server provides lots of primitives for managing graphical interfaces, together with many benefits? Let's just use it to get a canvas and reimplement everything ourselves inside that canvas (and thereby turn a snappy interface into something sluggish when the display is not local). * A GUI toolkit provides easy to use, multiplatform interface elements? Let's take a single program which only use a single canvas from the toolkit, and reimplement a whole GUI toolkit within (in one of the worst languages, over a program not made at all for the job, otherwise it wouldn't be fun). * The OS provides management for multiples users and groups, together with 2 systems to manage resources access rights? Nah, let's run everything as one user, and then try to implement some separation/protection within. Advanced level: run everything not only as a single user, but inside a single program, and then try to reimplement some isolation within it.
  - pdimitar 6y ago
    I agree with your premise but your examples lack nuance and aren't strictly true. > The OS provides process isolation and management, as well as various IPCs to deal with their interconnection? Let's use a single process and create threads, and then fibres. More performance? yes, a bit; more headaches? yes, many. Many one-off CLI script programs don't have the time to deal with N process forks (where N = CPU threads). This overhead is proven. It's objectively much quicker to spawn N threads inside the same process especially if your separate tasks need to converge and/or exchange messages here and there. And runtimes like Erlang's BEAM VM have unquestionable benefits (like preemptive green threads / fibers that exchange immutable messages), although they aren't well known for big raw processing muscle. > The display protocol/server provides lots of primitives for managing graphical interfaces, together with many benefits? Let's just use it to get a canvas and reimplement everything ourselves inside that canvas (and thereby turn a snappy interface into something sluggish when the display is not local). Which one is that? Linux's X11 is flawed in many regards, with the ability of each program to freely capture keystrokes being one of the most egregious. I can't blame people for trying to escape such hell and implement other platforms for the same thing -- without the drawbacks. > A GUI toolkit provides easy to use, multiplatform interface elements? Let's take a single program which only use a single canvas from the toolkit, and reimplement a whole GUI toolkit within (in one of the worst languages, over a program not made at all for the job, otherwise it wouldn't be fun). People wanted multi-platform desktop programs. F.ex. in my rather small country the only reason a software for managing shop stock and customer purchases, backorders, inventory etc. succeeded was because it was written in Java Swing; many shops used old Windows machines but no small amount of them also used Macs and some even used Ubuntu netbooks. That software would never sell back in 2007 if it wasn't OS-neutral. Also, Qt has commercial licensing that requires paying to use. Many, me included, don't want to deal with that. The programming world is complex as it is and if I suddenly find myself having to pay royalties, personally, to Qt, years after I stopped working for the employer I developed a desktop app for, that would be catastrophic. So many just dodge such potential bullets. > The OS provides management for multiples users and groups, together with 2 systems to manage resources access rights? Nah, let's run everything as one user, and then try to implement some separation/protection within. Advanced level: run everything not only as a single user, but inside a single program, and then try to reimplement some isolation within it. That's quite fair, however people want much more isolation than the OS usually offers -- hence stuff like FreeBSD jails, LXC containers / Docker, Windows' Sandboxie program etc. It's not enough to just run, say, Chrome, as a different user if you suspect Google is scanning your machine. --- Again, I agree with your premise but the examples aren't really accurate. A lot of the modern OS-es and and their technologies aren't as top-notch as we'd like (the Linux kernel and network infrastructure probably being some very rare exceptions).
- technofiend 6y ago
  Another vote for Parallel but I also wanted to mention using Redis for its atomic operations. It's very straight forward to push a list of things to do onto a redis queue and then have your scripts run via parallel pop one piece of work and then push the work item to either a success or failure queue. Unfortunately I no longer have a working example but as I recall it wasn't difficult to keep n number of scripts running and popping entries until the queue emptied.
  - pdimitar 6y ago
    Thanks for mentioning it. I gave up on Redis because I had a lot of trouble sharding it / distributing it years ago and I kind of forgot about it and stopped being interested in its features. But what you mention is very interesting and I'll have it in mind for the future. Thank you.
- gopalv 6y ago
  > The results were indeed embarrassingly parallel. The xargs/parallel mode can also scale up to many more nodes, if you are willing to introduce ssh into your shell scripts & do a minimum of stdio piping. I wrote a python mapreduce program named "batchman" a few years ago, which was aimed at running awk on hundreds of machines (& then being able to fire more things based on the results). zgrep on gzipped logs + awk + a central scheduler is extremely fast, to the point of beating Splunk to lookup exact things like ("does this zid have another session which they kept logging in while the game was throwing out-of-sequence errors?").
  - pdimitar 6y ago
    I have to admit I have no clue how to do that, any pointers? I thought of utilising my 4 machines at home sometimes for such things and would hugely benefit from implementing this using standard tools.
    - gobengo 6y ago
      gnu parallel can use cpus on remote hosts (via ssh) to parallelize work. https://www.gnu.org/software/parallel/parallel_tutorial.html#Remote-execution https://www.gnu.org/software/parallel/parallel_tutorial.html...
      - eythian 6y ago
        Well, that's interesting and useful to know. Thanks!
    - gopalv 6y ago
      > I have to admit I have no clue how to do that, any pointers? My code is poorly documented and badly written, but it works at least the ssh_copy_id one I use enough to keep updated[1] (that ssh logins with passwords to place the ssh keys). That's my script, the brains of it is basically an ssh connector. Paramiko is as simple as s = SSHClient() s.connect(hostname=h[1], username=h[0], password=pwd) (_in, _out, _err) = s.exec_command( cmd, bufsize=4096) Originally, I had awk scripts to process ^A separated, I'd read the out stream back and formulate my next script over the logs to drill-down in a loop. [1] - https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_id.py https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_...

pdimitar 6y ago

I have lately used a shell `for` loop that only emits part of the iterated values (not all are eligible for further processing) and then fed the loop directly to the `parallel` tool via the pipe operator. The results were indeed embarrassingly parallel. I am a fan of some languages that seem better equipped to utilise our modern many-core machines and I'd still write a longer-living system with better guarantees in those languages -- but many people ignore shell goodness at their own peril.

mturmon 6y ago
Big agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads. I've been developing a simulation software that does Monte Carlo over realizations of a simulated universe, and xargs and (later) parallel were super useful for parts of the workload. All the parallel job instances run the same simulation code, but for a different simulated universe, all controlled by a random number seed, so you can generate an ensemble of simulations with basically: head -NumberOfSimsWanted seeds.txt | xargs simulation.py
- pdimitar 6y ago
  Yep! Even as a fan of more parallel-friendly programming languages I still think having single-core algorithms applied to different input parameters in parallel is a very viable alternative approach if your problem space allows for it. I quite disliked the journey. The UNIX shell legacy shenanigans are real and can be a pain. But you can still bear a lot of fruit without going to the more arcane corners of it.
- flyinghamster 6y ago
  I wound up writing a hybrid bash script with embedded Makefile to re-encode my FLAC collection to Opus. It was nice to be able to use -j16 on that task. Switching to Opus saved a great deal of space on my phone's SD card.
- eythian 6y ago
  I have lately been working on a genetic algorithm system, written in single-threaded Perl. When I started off wanting to do, say, 20 simulations, I just used parallel, and it worked great on the 8 threads of my desktop machine. Now I've done a bit more and want to do long-running simulations that might take a few hours, I've taken to just firing it all at AWS Batch and having a script that uploads the results to S3. Now I can do 1-200 instances at the same time, which my own hardware isn't up to in a reasonable timeframe.
  - pdimitar 6y ago
    I have a pretty strong workstation (iMac Pro) that I currently don't use all the time. And a gaming PC that I barely touch in the last 9-10 months. Hit me up if you want me to lend you some CPU power sometime.
    - eythian 6y ago
      Cheers, but the configuring of it would be more hassle than it's worth I suspect. As it is, using AWS is very cheap as I can have it use spot instances. So hundreds of CPU-hours used for in the order of US$10-20.
      - pdimitar 6y ago
        I suspect setting that up would indeed be a hassle. :) Thanks for mentioning AWS Batch, I didn't know about it and will look it up.
qchris 6y ago
I've done something very similar recently, for processing hundreds of GB of .csv files across thousands of records. I ended up writing a fast single-threaded Rust program for the file parsing, then fed it into a bash script running GNU Parallel. That thing flies, and watching the system monitor showing every thread on the system get pegged to 100% simultaneously is a neat experience. I'm sure I could have done the parallelization itself in Rust given enough time, but honestly I found the `parallel` command to be so easy and resulting in so little overhead that it didn't really even seem worth it to spend the time on a language-native solution.
- spockz 6y ago
  Did the 100% system utilisation also lead to better performance? More cpu utilisation doesn’t always mean faster or higher throughput.
  - pdimitar 6y ago
    This can't be generalised but in many cases it did help me for CPU-bound tasks. Obviously there are cases where parallelisation is very superfluous. F.ex. if you have 1000 records of something, splitting the work of processing those (and we're talking really light processing like JSON encoding) among 4-8 threads is actually detrimental to performance.
  - qchris 6y ago
    That's certainly true, but it does look like in this case. I did the preliminary benchmarking on an 8-thread CPU (the actual HPC cluster has higher counts), and the total execution time on the processing was just under 8x, which I imagine came from the OS-level thread management overhead. I still need to check that pattern holds once thread counts increases, but I'm fairly optimistic it'll work out as long as I don't run into thermal throttling or I/O limits.
- heresie-dabord 6y ago
  GNU Parallel with awk is a SAPANS, effective combination for data analysis. No faddish rube-goldberg machines. https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/ *SAPANS: as simple as possible, and no simpler Do note that awk and GNU Parallel are tools that use/contain C-like syntax, which is abhorrent to some factions. GNU Parallel is written in Perl.
wott 6y ago
At lot of people do not like to use the features already provided by their environment and like to take a single element of the original features set and reimplement the same features within that single element. Sometimes it improves performance, sometimes not. Sometimes it improves productivity, sometimes not. I am not sure of the global balance. * The OS provides process isolation and management, as well as various IPCs to deal with their interconnection? Let's use a single process and create threads, and then fibres. More performance? yes, a bit; more headaches? yes, many. * The display protocol/server provides lots of primitives for managing graphical interfaces, together with many benefits? Let's just use it to get a canvas and reimplement everything ourselves inside that canvas (and thereby turn a snappy interface into something sluggish when the display is not local). * A GUI toolkit provides easy to use, multiplatform interface elements? Let's take a single program which only use a single canvas from the toolkit, and reimplement a whole GUI toolkit within (in one of the worst languages, over a program not made at all for the job, otherwise it wouldn't be fun). * The OS provides management for multiples users and groups, together with 2 systems to manage resources access rights? Nah, let's run everything as one user, and then try to implement some separation/protection within. Advanced level: run everything not only as a single user, but inside a single program, and then try to reimplement some isolation within it.
- pdimitar 6y ago
  I agree with your premise but your examples lack nuance and aren't strictly true. > The OS provides process isolation and management, as well as various IPCs to deal with their interconnection? Let's use a single process and create threads, and then fibres. More performance? yes, a bit; more headaches? yes, many. Many one-off CLI script programs don't have the time to deal with N process forks (where N = CPU threads). This overhead is proven. It's objectively much quicker to spawn N threads inside the same process especially if your separate tasks need to converge and/or exchange messages here and there. And runtimes like Erlang's BEAM VM have unquestionable benefits (like preemptive green threads / fibers that exchange immutable messages), although they aren't well known for big raw processing muscle. > The display protocol/server provides lots of primitives for managing graphical interfaces, together with many benefits? Let's just use it to get a canvas and reimplement everything ourselves inside that canvas (and thereby turn a snappy interface into something sluggish when the display is not local). Which one is that? Linux's X11 is flawed in many regards, with the ability of each program to freely capture keystrokes being one of the most egregious. I can't blame people for trying to escape such hell and implement other platforms for the same thing -- without the drawbacks. > A GUI toolkit provides easy to use, multiplatform interface elements? Let's take a single program which only use a single canvas from the toolkit, and reimplement a whole GUI toolkit within (in one of the worst languages, over a program not made at all for the job, otherwise it wouldn't be fun). People wanted multi-platform desktop programs. F.ex. in my rather small country the only reason a software for managing shop stock and customer purchases, backorders, inventory etc. succeeded was because it was written in Java Swing; many shops used old Windows machines but no small amount of them also used Macs and some even used Ubuntu netbooks. That software would never sell back in 2007 if it wasn't OS-neutral. Also, Qt has commercial licensing that requires paying to use. Many, me included, don't want to deal with that. The programming world is complex as it is and if I suddenly find myself having to pay royalties, personally, to Qt, years after I stopped working for the employer I developed a desktop app for, that would be catastrophic. So many just dodge such potential bullets. > The OS provides management for multiples users and groups, together with 2 systems to manage resources access rights? Nah, let's run everything as one user, and then try to implement some separation/protection within. Advanced level: run everything not only as a single user, but inside a single program, and then try to reimplement some isolation within it. That's quite fair, however people want much more isolation than the OS usually offers -- hence stuff like FreeBSD jails, LXC containers / Docker, Windows' Sandboxie program etc. It's not enough to just run, say, Chrome, as a different user if you suspect Google is scanning your machine. --- Again, I agree with your premise but the examples aren't really accurate. A lot of the modern OS-es and and their technologies aren't as top-notch as we'd like (the Linux kernel and network infrastructure probably being some very rare exceptions).
technofiend 6y ago
Another vote for Parallel but I also wanted to mention using Redis for its atomic operations. It's very straight forward to push a list of things to do onto a redis queue and then have your scripts run via parallel pop one piece of work and then push the work item to either a success or failure queue. Unfortunately I no longer have a working example but as I recall it wasn't difficult to keep n number of scripts running and popping entries until the queue emptied.
- pdimitar 6y ago
  Thanks for mentioning it. I gave up on Redis because I had a lot of trouble sharding it / distributing it years ago and I kind of forgot about it and stopped being interested in its features. But what you mention is very interesting and I'll have it in mind for the future. Thank you.
gopalv 6y ago
> The results were indeed embarrassingly parallel. The xargs/parallel mode can also scale up to many more nodes, if you are willing to introduce ssh into your shell scripts & do a minimum of stdio piping. I wrote a python mapreduce program named "batchman" a few years ago, which was aimed at running awk on hundreds of machines (& then being able to fire more things based on the results). zgrep on gzipped logs + awk + a central scheduler is extremely fast, to the point of beating Splunk to lookup exact things like ("does this zid have another session which they kept logging in while the game was throwing out-of-sequence errors?").
- pdimitar 6y ago
  I have to admit I have no clue how to do that, any pointers? I thought of utilising my 4 machines at home sometimes for such things and would hugely benefit from implementing this using standard tools.
  - gobengo 6y ago
    gnu parallel can use cpus on remote hosts (via ssh) to parallelize work. https://www.gnu.org/software/parallel/parallel_tutorial.html#Remote-execution https://www.gnu.org/software/parallel/parallel_tutorial.html...
    - eythian 6y ago
      Well, that's interesting and useful to know. Thanks!
  - gopalv 6y ago
    > I have to admit I have no clue how to do that, any pointers? My code is poorly documented and badly written, but it works at least the ssh_copy_id one I use enough to keep updated[1] (that ssh logins with passwords to place the ssh keys). That's my script, the brains of it is basically an ssh connector. Paramiko is as simple as s = SSHClient() s.connect(hostname=h[1], username=h[0], password=pwd) (_in, _out, _err) = s.exec_command( cmd, bufsize=4096) Originally, I had awk scripts to process ^A separated, I'd read the out stream back and formulate my next script over the logs to drill-down in a loop. [1] - https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_id.py https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_...