10 ms·
Parallel – A command-line CPU load balancer written in Rust
- mnkmnk 10y agoThis looks like a perfect real world complete useful application that a beginner in rust could look at.
- 0xmohit 10y agoA rust beginner could also look at Jonathan Turner's lessons from solving first 12 Project Euler problems. [0] https://www.jonathanturner.org/2015/10/lessons-from-first-12-euler-problems-in.html https://www.jonathanturner.org/2015/10/lessons-from-first-12...
- eriknstr 10y ago>The owner of www.jonathanturner.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website. I added an exception as I pretty much always do but in case the owner of the site is around here, they might want to investigate this.
- steveklabnik 10y agoI've just pinged them, thanks.
- jntrnr 10y agowebsite owner here: odd, what configuration of firefox gives you that warning. When I connect with a recent OS X version, it seems to work okay.
- sp332 10y agoFirefox on Windows here. Looks like it's getting the certificate for github, but throwing an error because your domain isn't listed in that certificate. https://imgur.com/a/UeXtC https://imgur.com/a/UeXtC
- jntrnr 10y agoYeah, it's not set up for https. Do you see the same thing with http://www.jonathanturner.org/ http://www.jonathanturner.org/?
- sp332 10y agoNope, that's fine. Not sure why 0xmohit posted the HTTPS link in the first place.
- heinrich5991 10y agoSame error for Firefox on Linux.
- zhanxw 10y agoWhy this can be faster than GNU parallel?
- mcpherrinm 10y agoOne possible reason is that GNU parallel is a perl script.
- ploxiln 10y agoa version in C that I think was first released before GNU parallel is in "moreutils" https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/ A few years ago, debian made GNU parallel provide the "/usr/bin/parallel" executable, instead of moreutils. The maintainer of moreutils had some interesting things to say about that: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050#75 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050#75
- paulddraper 10y ago> It is 5143 lines of code, and the anthethesis of a simple, clean, well-designed unix command. Only 5k? :D But, yes, the criticisms are valid. I recommend moreutils.
- dagw 10y agoHe lost me at the point when he complained that GNU parallel "includes the ability to transfer files between computers". For me at least that is _the_ feature of GNU parallel that actually makes it really useful. Which I guess is the problem with all these discussions, one persons useless bloat is another persons nr. 1 killer feature.
- oblio 10y agoBut in the spirit of Unix, shouldn't parallel exec some file transfer program?
- bmalehorn 10y agoThe code is more complicated than it needs to be. It spawns N threads, then each thread forkexecs child processes. These threads communicate through an atomic int. There is no need for threads. Just spawn background processes: echo 1 & echo 2 & wait echo 3 & wait echo 4 & ... The key is that the wait() system call will hang until any child process finishes.
- the_mitsuhiko 10y agoI doubt that will make it measurable more efficient. Threads are cheap.
- bmalehorn 10y agoMy point wasn't optimization. Just that the author was making it more complicated than was needed. It's like discovering that "ls" is spawning 5 threads for internal communication.
- the_mitsuhiko 10y agoIn a language with good threading support, internal threads are typically making things easier rather than harder. I tend to spawn lots of threads in Rust just because I can and it simplifies the code a lot over having to do some async callback mess. In particular there is no sane way to async waitpid() on POSIX.
- geofft 10y agopselect / self-pipe / etc. on SIGCHLD, and on receipt of SIGCHLD, loop through all known child processes with wait(WNOHANG). ... I'm not sure I can disagree with "no sane way".
- the_mitsuhiko 10y agoThe only thing you can do from a signal handler is flipping a static and only one signal handler can be set. Good luck making this reusable. In fact, I challenge you to solve the problem "spawn process; waitpid for 15 seconds; otherwise kill hard" in Rust (or C++ if you feel like) on POSIX once with threads and once without threads by sticking to what's permitted in the standard and so that multiple processes can be waited for. Then also measure CPU impact :)
- Aissen 10y agoNot comparing it with the simpler, better moreutils parallel is really unfair.
- ole_tange 10y agoI hope you wont take this the wrong way, but can you elaborate on why you feel Tollef's parallel (from moreutils) is better than GNU Parallel?
- Aissen 10y agoIt was addressed elsewhere in the thread. My take is: 1/ it's written in C. 2/ it doesn't suffer from feature creep. The code is 427 lines of C. GNU parallell is 10k+ lines of perl. Considering mmstick compares the loading times, it's easy to see where the difference comes from.
- kwoff 10y agoLike headlines that are questions (answer: "no"; next), I tend to skip over headlines that say "written in $language". Usually means they've copied something that already exists, or it doesn't have much else distinctive about it than being written in that language.
- sopium 10y agoI often use make to run jobs in parallel. It is nice to be able to continue after an error or breakage, especially when working with large workloads.
- ole_tange 10y agoUsing Makefiles, however, depends on the results being files. If that is not the case, make cannot tell how far you got. GNU Parallel has --joblog and can continue from where it left off or retry all failed jobs again.
- RhysU 10y agoFurther overloading this overloaded utility name screws up packagers and hinders usage of any of the variants. Rename it please.
- gcr 10y agoAgreed. GNU Parallel does something similar and this is confusing. But this does claim to be a reimplementation of much of GNU Parallel's functionality. If it's compatible, maybe there isn't too much of an issue. In the interest of silly UNIXism, I hereby vote for the name "serial" instead. It's just like "more" and "less", right?
- kibwen 10y agoI believe it intends to support a subset of GNU parallel and be command-line compatible with that subset. And "perpendicular" would be a far better joke name, IMO. :P
- thealistra 10y agoI always tended to autogenerate a makefile with fake goals like: g1: job1 g2: job2 and then one goal to depend on them all and I used make -j 4
- kstrauser 10y agoparallel is awesome for when you want to want to run a number of jobs unknown in advance, such as using it as an xargs replacement: find . -type f -name | parallel --jobs 5 process_file That will run "process_file foo" for ever file in a directory with 5 processes running in parallel.
- coredog64 10y agoNot being a jerk, but what does parallel bring to the table that xargs doesn't? find . -type f -name | xargs -I {} -n 1 -P 5 process_file {}
- perturbation 10y agoYou can have remote workers with parallel (not sure if you can do that with xargs or not). parallel --progress --wd '.' -S '4/fourworkers@blehworker.org,8/eightworkers@eightworkers.org,2/:' "./complicated_command --param1={1} --param2={2} ::: $(some_shellcommand_to_generate_param1_values.sh) ::: $(some_shellcommand_to_generate_param2_values.sh) It's very flexible, become my go-to for speeding up parallelizable work. Next step up would be Celery or Spark or something.
- ole_tange 10y agoBigger list: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
- ole_tange 10y agoFrom: https://www.gnu.org/software/parallel/history.html https://www.gnu.org/software/parallel/history.html > parallel dates back to around the same time. It was originally a wrapper that generated a makefile and used make -j to do the parallelization.
- amattn 10y agoI wrote a go utility that is not designed to be direct replacement, but rather nicer CLI tool that does something quite similar. My version is quite a bit more limited, but totally meets my current needs. https://github.com/amattn/paral https://github.com/amattn/paral