3 ms·
GNU/Parallel changed my life
- praptak 16y ago"Unlike xargs however, Parallel lets me make use of the many cores that I have access to [...]" Unlike xargs? That's just plain wrong, see xargs' -P flag.
- wazoox 16y agoxargs can't distribute tasks to remote CPUs though.
- chrisaycock 16y agoPrevious discussion on GNU parallel: http://news.ycombinator.com/item?id=1801186 http://news.ycombinator.com/item?id=1801186
- ma2rten 16y agoOh man ! I wish someone had told me this before. Back when i did research for my BSc thesis, I had an 24 core machine and wrote a script to create 24 bash scripts to split up my task which i would than run in parallel. Sometimes I have the felling that I am wasting time when I check HN too often, but stuff like this makes me think otherwise.
- chrisaycock 16y agoYou could have also used "make -j", depending on your specific task.
- fish2000 16y agoThis is sometimes true. I think that Parallel is more interesting as an instant server-farm for "taco bell programmers" -- q.v. http://teddziuba.com/2010/10/taco-bell-programming.html http://teddziuba.com/2010/10/taco-bell-programming.html and http://news.ycombinator.com/item?id=1818816 http://news.ycombinator.com/item?id=1818816 for the HN discussion -- as per Mr. Dziuba's definition. Like, async spidering with parallel is a one-liner, but comparatively, have you seen how much absurdly syntaxed pycurl boilerplate you need? It's crazy. I for one will take the simpler method any day.
- JulianMorrison 16y agoThat is just going to thrash hell out of your disk unless it's executing on distinct machines with data well spread over a shared filesystem.
- olliesaunders 16y ago"Changed my life," a bit dramatic methinks.