4 ms·
Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including,
by sethammons 3y ago
Structured logging is my jam. Our unit tests validate error content, errors have all dynamic text in different fields, provide all info to reproduce (including, where appropriate, copy-pastable curl commands to replicate an error to remote api endpoints!) and log aggregators like Splunk can group and aggregate and find relationships.
If I see a spike in errors, I can, in real time, count them by user id, group by region, filter out by some other field, then have the table of events generate a live line chart easily showing the error rates over time, and pin pointing the exact moment an error started.
Less than five to ten minutes after detection or report and I can be talking with another team about a change they released at time $x, including a list of affected users for product teams to reach out to if needed.
- junon 3y agoDoesn't the curl replication leak secrets, though?
- sethammons 3y agoDepends on what you are doing. We mask auth stuff and some known params like an auth header might log as "authorization: bearer $APIKEY". Some things can't be logged. But as a general rule, it is an immense time saver.
- cogman10 3y agoYeah, structured logging has a tendency to do that as well. Logs are often a treasure trove for system invaders.
- hallman76 3y agoThis sounds awesome - pretty disk-heavy but worth it. How long do you retain these logs? What sort of user/request volume do you deal with? Do you have any resources/blog posts/keywords for learning more?
- jonhohle 3y agoFor compressed logs, it doesn’t have to be significant. If you can get structured logs into a TSDB you really only need to retain logs in cold storage or for as long as more detailed views or correlation might be necessary.
- sethammons 3y agoSendGrid, so billions of incoming requests that multiplex out to at a minimum 8 other log events, plus all errors, etc. We have to run our own splunk instances. We used to store data for a much longer time but as our scale keeps going up so do the costs. We've had to reduce this to a 7 day lookback for higher volume services. For lower volume (in the millions not billions), 30 days to a year lookback depending. As for blog posts - I'm not aware of any. I've actually wanted to show off what we have but have never prioritized the blog post.
- b0afc375b5 3y ago> As for blog posts - I'm not aware of any. I've actually wanted to show off what we have but have never prioritized the blog post. Please do, I'm interested in learning as well.
- deleted 3y ago[deleted]
- iman453 3y agoThis sounds awesome. Could you share some examples of logs you have?