4 ms·
> The results were indeed embarrassingly parallel. The xargs/parallel mode can also scale up to many more nodes, if you are willing to introduce ssh into your
by gopalv 6y ago
> The results were indeed embarrassingly parallel.
The xargs/parallel mode can also scale up to many more nodes, if you are willing to introduce ssh into your shell scripts & do a minimum of stdio piping.
I wrote a python mapreduce program named "batchman" a few years ago, which was aimed at running awk on hundreds of machines (& then being able to fire more things based on the results).
zgrep on gzipped logs + awk + a central scheduler is extremely fast, to the point of beating Splunk to lookup exact things like ("does this zid have another session which they kept logging in while the game was throwing out-of-sequence errors?").
- pdimitar 6y agoI have to admit I have no clue how to do that, any pointers? I thought of utilising my 4 machines at home sometimes for such things and would hugely benefit from implementing this using standard tools.
- gobengo 6y agognu parallel can use cpus on remote hosts (via ssh) to parallelize work. https://www.gnu.org/software/parallel/parallel_tutorial.html#Remote-execution https://www.gnu.org/software/parallel/parallel_tutorial.html...
- eythian 6y agoWell, that's interesting and useful to know. Thanks!
- gopalv 6y ago> I have to admit I have no clue how to do that, any pointers? My code is poorly documented and badly written, but it works at least the ssh_copy_id one I use enough to keep updated[1] (that ssh logins with passwords to place the ssh keys). That's my script, the brains of it is basically an ssh connector. Paramiko is as simple as s = SSHClient() s.connect(hostname=h[1], username=h[0], password=pwd) (_in, _out, _err) = s.exec_command( cmd, bufsize=4096) Originally, I had awk scripts to process ^A separated, I'd read the out stream back and formulate my next script over the logs to drill-down in a loop. [1] - https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_id.py https://github.com/t3rmin4t0r/batchman/blob/master/ssh_copy_...