4 ms·
I regularly search logs across a huge fleet of hosts (thousands). I have a script that will send an arbitrary command in parallel to each host and return the o
by cle 12y ago
I regularly search logs across a huge fleet of hosts (thousands). I have a script that will send an arbitrary command in parallel to each host and return the output in a JSON file.
Once I get the logs I'm interested in, it's usually a straightforward combination of jq, grep, sed, awk, cut, sort, uniq, xargs, etc. If I need to do some fancy queries on the data, I have another script that will parse the logs and load them into a SQLite db.
- brown9-2 12y agoWhat are you using to execute commands on thousands of hosts in parallel?
- thrownaway2424 12y agoI don't know what the GP's answer will be, but I would use pdsh.
- cle 12y agoJust forking off processes that SSH into the hosts and write to temp files, which are picked up the by the main process. It's gotten pretty elaborate, but the basics are pretty simple.