5 ms·
Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep
by dsXLII 3y ago
Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.
- vincnetas 3y ago100GB of logs per day? what kind of applications are that chatty?
- getrealyall 3y agoAnd I bet a hang glider can't fly from New York to Paris, either! The nerve! Recall that the poster said this was for a small startup. If you're Google, by all means, use Google logging tools. If you aren't, then solve the problem you have, not the problem your résumé needs.
- phyrex 3y agoThe guy asked > Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog a the like. And the poster above answered that question
- 10000truths 3y agoThey scale perfectly fine, as long as you filter locally before aggregating. Lo and behold: mkdir -p /tmp/ssh_output while read ssh_host; do ssh "$ssh_host" grep 'keyword' /var/my/app.log > "/tmp/ssh_output/${ssh_host}.log" & done < ssh_hosts.txt wait cat /tmp/ssh_output/*.log rm -rf /tmp/ssh_output Tweak as needed. Truncation of results and real-time tailing are left as an exercise to the reader.