2 ms·
> "for example by running a script on their server logs" Exactly! There's lots of stuff you can do before you write a $1M check to one of these APM SaaS. Time
by steveBK123 3y ago
> "for example by running a script on their server logs"
Exactly! There's lots of stuff you can do before you write a $1M check to one of these APM SaaS.
Timestamp each hop. Do your percentiles on a trailing X day window each morning. Alert on worrying moves in the p99/95/whatever.
Make sure you don't run out of disk/memory/database by like.. alerting when they get to 80/90/whatever %. Amazing footgun to not monitor this simple thing.
Have a process that serves as a reasonable proxy for "busy-ness" / bottlenecks. ie - gateways,
Monitor the % CPU over X minute windows.
If the CPU stays saturated for N minutes, something is bottlenecking.
Here's another secret - you want to also monitor for the opposite. Have processes that are "workers" of some sort that should always have load during business hours as there is always data flowing through? Monitor that their CPU doesn't fall BELOW some threshold (it means something fell over and they aren't doing work).
None of this has to alert within milliseconds.
Simply having it, and it alerting within a few minutes is better than not having it.
Some of these things like latency trends can be run nightly, whatever.
That's an interns work for a week or a seniors afternoon.
This is easy on s5s (servers). I'm sure you can do this in k8s or whatever too.