Here is a question, I mean it honestly, I'm relatively old school and have built many apps using syslog. When it comes to doing log mining, I've got a fairl

Here is a question, I mean it honestly, I'm relatively old school and have built many apps using syslog. When it comes to doing log mining, I've got a fairly old school utility belt, I poke around with less, I cat through grep (really ripgrep), I cat through grep and pipe to awk and extract things. Sometimes I fire up cut. I get a ton of milage from sort and uniq. Obviously, I fire up zcat in place of cat when needed. I also generously apply find when needed. It feels like I find what I need pretty quickly. Admittedly, I generally don't have terabytes of logs with these tools but it handles 10s of GB shockingly well. With Splunk, ELK, Greylog, it feels insanely pokey. I know they have the parsers and such. At times I've kind of boned up on their search syntax but I've never gone "all in" with any of them, maybe because they all don't seem like a really solid long term solution. They seems to have a different kind of model than what I want, the time range is kind of nice but often times I won't have a time range until later. My model involves winnowing down the the data I want and then extracting pieces and viewing the data different ways. Am I just using all these tools the wrong way? Is my mental model off? Maybe it's a log consistency thing, it's always sort of a great day when you get "Error: abc failed because xyz and def." and that's the answer to everything. Many times I'll be spending time looking at logs and I'll notice an increase in a certain behavior happened before the outage happened and that's the give away.. Then a new grafana dashboard is created with a new metric to try and identify that before it happens again. Loki kind of looks like it supports my method but again, I'm back to that "I haven't gone all in" with it problem. As I'm rambling, I've seen these sexy dashboards with like red/yellow/green lights and some latency graphs and cool looking stuff and then a little table of the last 20 "log messages" and maybe I'm used to looking at logs that you don't show in your dashboard or something like that. They all feel like a square hole to my round peg. Maybe it's just me.