4 ms·

Big agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads. I've been developing a simulation software

Big agree. Gnu parallel, xargs, and make -j are all very useful basic tools for embarrassingly parallel workloads. I've been developing a simulation software that does Monte Carlo over realizations of a simulated universe, and xargs and (later) parallel were super useful for parts of the workload. All the parallel job instances run the same simulation code, but for a different simulated universe, all controlled by a random number seed, so you can generate an ensemble of simulations with basically: head -NumberOfSimsWanted seeds.txt | xargs simulation.py

pdimitar 6y ago
Yep! Even as a fan of more parallel-friendly programming languages I still think having single-core algorithms applied to different input parameters in parallel is a very viable alternative approach if your problem space allows for it. I quite disliked the journey. The UNIX shell legacy shenanigans are real and can be a pain. But you can still bear a lot of fruit without going to the more arcane corners of it.
flyinghamster 6y ago
I wound up writing a hybrid bash script with embedded Makefile to re-encode my FLAC collection to Opus. It was nice to be able to use -j16 on that task. Switching to Opus saved a great deal of space on my phone's SD card.
eythian 6y ago
I have lately been working on a genetic algorithm system, written in single-threaded Perl. When I started off wanting to do, say, 20 simulations, I just used parallel, and it worked great on the 8 threads of my desktop machine. Now I've done a bit more and want to do long-running simulations that might take a few hours, I've taken to just firing it all at AWS Batch and having a script that uploads the results to S3. Now I can do 1-200 instances at the same time, which my own hardware isn't up to in a reasonable timeframe.
- pdimitar 6y ago
  I have a pretty strong workstation (iMac Pro) that I currently don't use all the time. And a gaming PC that I barely touch in the last 9-10 months. Hit me up if you want me to lend you some CPU power sometime.
  - eythian 6y ago
    Cheers, but the configuring of it would be more hassle than it's worth I suspect. As it is, using AWS is very cheap as I can have it use spot instances. So hundreds of CPU-hours used for in the order of US$10-20.
    - pdimitar 6y ago
      I suspect setting that up would indeed be a hassle. :) Thanks for mentioning AWS Batch, I didn't know about it and will look it up.

pdimitar 6y ago

Yep! Even as a fan of more parallel-friendly programming languages I still think having single-core algorithms applied to different input parameters in parallel is a very viable alternative approach if your problem space allows for it. I quite disliked the journey. The UNIX shell legacy shenanigans are real and can be a pain. But you can still bear a lot of fruit without going to the more arcane corners of it.

flyinghamster 6y ago

I wound up writing a hybrid bash script with embedded Makefile to re-encode my FLAC collection to Opus. It was nice to be able to use -j16 on that task. Switching to Opus saved a great deal of space on my phone's SD card.

eythian 6y ago

I have lately been working on a genetic algorithm system, written in single-threaded Perl. When I started off wanting to do, say, 20 simulations, I just used parallel, and it worked great on the 8 threads of my desktop machine. Now I've done a bit more and want to do long-running simulations that might take a few hours, I've taken to just firing it all at AWS Batch and having a script that uploads the results to S3. Now I can do 1-200 instances at the same time, which my own hardware isn't up to in a reasonable timeframe.

pdimitar 6y ago
I have a pretty strong workstation (iMac Pro) that I currently don't use all the time. And a gaming PC that I barely touch in the last 9-10 months. Hit me up if you want me to lend you some CPU power sometime.
- eythian 6y ago
  Cheers, but the configuring of it would be more hassle than it's worth I suspect. As it is, using AWS is very cheap as I can have it use spot instances. So hundreds of CPU-hours used for in the order of US$10-20.
  - pdimitar 6y ago
    I suspect setting that up would indeed be a hassle. :) Thanks for mentioning AWS Batch, I didn't know about it and will look it up.