3 ms·
> And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a
by jedberg 10y ago
> And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a significant difference between the two.
That's an important distinction, one that I agree with, and I should make clearer.
Logs do have a purpose, but I'm not sure that retaining them does.
Sure, for a very small shop, throw them on a disk, use awk, sed, grep, and perl to look through them, and call it a day. But once you get to the point of "spinning up a cluster of log servers" or something like it, I'd say you're probably better off investing in monitoring instead.
- jsmthrowaway 10y agoHaving a corpus of live "test" requests for rebuilding an API is a compelling enough use case to justify infinite retention, in my opinion. Perhaps relevantly to your prior gig: "did we handle that weird Sony Bravia API call from five years ago in this redesign? Yes we did." I'm implying a little more structure to logging, though, because it sounds like you're thinking about random strings emitted from binaries (which I think are also approachable, to an extent). This overlaps with PII and such, though, and there is no one-size-fits-all answer. More than once I've run the entire corpus of requests to a system, ever, through a dummy rebuild as a pretty great integration test. It's a powerful SRE tool. Spelunking through all historical data is just icing on that cake, honestly. As the author says, SRE is basically just an information factory; I'd be haaaaaard pressed to agree with you on throwing away a lot of information -- you don't know what you don't know until you want to know it -- and betting all-in on monitoring. Retaining logs is not the hardest problem SRE deals with, either, but SREs turn around and force unrealistic latency requirements on the query side (I see a lot of ELK deploys running into this). You have to look at it as an Oracle. Oh, great Oracle of a pile of meaningless logs, cook off this map/reduce and tell me an interesting number that I can put in a Keynote for executives. Definitely not dashboarding from logs data. That's an impedance mismatch that Google gets away with because of the nature of their logging.
- user5994461 10y agoSplunking is cool. Nonetheless, splunk is the most expensive software license on the planet. More than Oracle, yes.
- jsmthrowaway 10y agoI meant spelunking, where they got their name. But yes, you're not wrong. (I've heard of $millions.)
- zwischenzug 10y agoSome envs require logs to be kept for regulatory purposes. Interestingly, we never used a cluster of log servers. I was always skeptical of their utility. It was grep, plus some hand-rolled utility scripts to interrogate. One was a thing of beauty I spent years on, which saved us a ton of time.