4 ms·
One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem
by arp242 4y ago
One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean.
An error is either:
1. A real bug someone ran in to.
2. Not a bug and just a spurious "error", which should be a log message, some statistic somewhere, or just disabled outright. Errors should only be errors – an uncaught exception, explicit error log, stuff like that. Everything else should be something else.
What I've found – and this certainly won't hold true for all cases, but it does seem to be the case often – is that half or more all in the second category: it's just stuff that shouldn't be logged as an error. This is usually trivial to fix.
This still leaves you with 5,000 errors/day, but typically ~10 bugs will cause ~4,950 of them. This takes more time to fix, depending on the bug, but usually also takes less time than "5,000 errors every day!" makes it sound.
The last ~50 errors are usually the hardest, especially if it's frontend JavaScript stuff. I "read every error", but I have a few that pop up regularly that I just can't reproduce and don't know how to fix. I think it may be extensions people use or something, but idk. Since this post talks about APIs I guess that's not really applicable to them.
- skipkey 4y ago> One thing that I've observed is that the more errors there are, the less likely people will look at them It can be even worse than that. At one point I worked on an app that generated north of 500 lines in the PHP error logs per minute per server. In the course of implementing a feature I cleaned up a few of the low hanging warnings near the code I was modifying. This resulted in about a 3 percent reduction in the log file velocity. When we deployed to prod, everyone panicked, because log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health. That was the one and only time in my career I was reprimanded for cleaning up warnings.
- arp242 4y ago> log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health I don't want to bash on PHP like it's 2002, but ... that's probably one of the most PHP things I ever heard. I worked somewhere where people were against raising error_reporting because someone was in the habit of writing "flase" instead of "false" – which worked because PHP – and it wasn't deemed worth fixing :-/ Old PHP days were wild.
- blackflame7000 3y ago[dead]
- MichaelZuo 3y ago'log file velocity' ... that's one way to go about it!
- Gigachad 4y agoA good logging tool will group similar errors together so one error occurring 10,000 times shows once in the list. But what I’ve found is a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve.
- arp242 4y agoGrouping helps, but often errors are distinct enough that they will get duplicated regardless, especially after updates which changes the stack trace. To be honest it's been a few years since I used these tools, but last time it was helpful but very much less-than-perfect. > a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve. Depends what is timing out; if it's SQL queries then you should probably do something somewhere, if it's HTTP connections then you probably shouldn't log them as an error but as info/debug, keep some aggregate statistic somewhere, or just not do anything with it at all (depending on your scale and what you need).
- lanstin 4y agoOr switch to gRPC :) if it is internal to the data center traffic.
- AtNightWeCode 3y agoMost logging tools if used correctly separates static messages from dynamic values. Which makes grouping trivial.
- RheingoldRiver 4y ago> One thing that I've observed is that the more errors there are, the less likely people will look at them And then you have nodejs warnings
- eyelidlessness 3y ago> One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean. If I may broaden this, I think the same principle applies for: - poor performance - tech debt/otherwise hard to manage extant functionality - gaps in documentation And probably a lot more. If it’s one or a few narrowly identifiable problems, it can be easy to escalate and rally and solve. If it feels pervasive, even trying to address one thing feels insurmountable. And speaking to this from a perspective of being highly motivated to tackle problems like those I mention, and having some fairly significant successes in that regard over recent months, it still feels daunting. But it feels a lot more reasonably possible and more importantly demonstrably valuable the more I chip away at it. But it has taken actual years to get even that far.