6 ms·
A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different servi
by hliyan 3y ago
A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in C++, each producing over 1GB of logs each day. Managed to debug some fairly complex algorithmic trading bugs. Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog an the like.
- dsXLII 3y agoVolume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale especially well.
- vincnetas 3y ago100GB of logs per day? what kind of applications are that chatty?
- getrealyall 3y agoAnd I bet a hang glider can't fly from New York to Paris, either! The nerve! Recall that the poster said this was for a small startup. If you're Google, by all means, use Google logging tools. If you aren't, then solve the problem you have, not the problem your résumé needs.
- phyrex 3y agoThe guy asked > Twenty years later, I still can't fathom why we're spending so much money on Splunk, DataDog a the like. And the poster above answered that question
- 10000truths 3y agoThey scale perfectly fine, as long as you filter locally before aggregating. Lo and behold: mkdir -p /tmp/ssh_output while read ssh_host; do ssh "$ssh_host" grep 'keyword' /var/my/app.log > "/tmp/ssh_output/${ssh_host}.log" & done < ssh_hosts.txt wait cat /tmp/ssh_output/*.log rm -rf /tmp/ssh_output Tweak as needed. Truncation of results and real-time tailing are left as an exercise to the reader.
- getrealyall 3y agoFinancialization and mediocre developers. I haven't worked with too many people I could actually trust to even emit logs correctly, let alone develop a tool to collect and aggregate them. I've also been told, time and again, in no uncertain terms, to "buy as much as possible". We've reached the logical conclusion of SaaS-everything: every company just cobbles together expensive, overcomplicated computers from other expensive, overcomplicated computer providers, resulting in expensive, bloated systems that barely work.
- diarrhea 3y agoBuying everything and SaaSing the whole place up is a true killjoy. I giggle with joy whenever I am allowed to write code. And then a support request comes in that I get assigned to, “thing in SaaS doesn’t work please fix”. And all you have to debug that SaaS is their UI. The checkbox in question is on, you notice, so it can only be a bug on their side. Off to contacting support as the only available avenue. Incredibly boring.
- quickthrower2 3y agoYay I get to write code! Oh it’s Pulumi code that wraps terraform to deploy Elastic Search and configure networking to allow logs from a kubernetes we deployed the same way.
- fragmede 3y agoWhat did you use for visualization in that stack? The fact that I can "|" (pipe) my data and make bar and pie charts is what really does it for me. What's really money is trivially being able to see requests coming in overlaid on a world map. I was sold the first time I saw that because it let me fix an issue that would have taken me hours to suss out just grepping around. More power to you for using sed awk and grep, they're powerful tools and every computer person should know how to use them. But if you're hung up on only using sed awk and grep for emotional reasons, that's self-limiting. We have better tools today, and you don't get hero points for using shittier tools when there are better ones available to you. https://www.splunk.com/en_us/blog/tips-and-tricks/mapping-with-splunk.html https://www.splunk.com/en_us/blog/tips-and-tricks/mapping-wi...