8 ms·
Read Every Single Error
- jacurtis 4y agoThis article comes off as "holy than thou". While maintaining what seems like ignorance in the "haute culture" that they mock. Their primary point of "read every error" is one that almost no one would truly debate (within reason). Most companies do try to read every error as is reasonable. Being aware of an error is different than immediately fixing it (since that can usually be impractical). Where companies tend to differ is in their ability to triage and correct errors. That is the secret sauce that people want to hear. One of their secondaries points is mocking companies that use error system aggregators. Why is this so bad? They claim it prevents you from reading every error, but I'd beg to differ. Let's say your system gets 5,000 errors today. It is highly likely that many of those errors are replicate instances of the same error. Out of 5,000 errors you probably have 5 errors that are 800-1,000 instances each, which consumes 4,800 or so of the error instances. Then you probably have a long tail of errors that only show up once or twice a day. So why is an error aggregator bad? Is there any point in reading the same error 1,000 times? Wouldn't it make more sense to read the error once, have the system show you that it shows up 1,000 which is the highest in your system. Then you can spend less time reading more duplicate errors and spend that time fixing them. You can look at a graph and see that the error increases over time, which tells you to prioritize resolution of this error. If you have 2 hours a day to fix bugs, would you rather fix the bug that causes 2 errors per day or the one that fixes 1,000? They seem proud of their error burndown chart that shows errors decreasing over time, but this is likely not the result of "reading every error" as much as actually putting time and attention into error resolution. My company for example put a team together at the end of last year who is a cross-functional team who's sole job is fixing prominent errors. We use an error aggregator and our error burndown chart looks the same as Pulumi's. Why? Because we are dedicating resources and attention to fixing it. Where most companies go wrong is not that they don't read errors. It is that they are so busy pumping out features that they don't dedicate time and resources to fixing things or they don't provide enough time to build less error-prone code before it hits production in the first place. Lastly, the "haute culture" of error budgets are funny to me. Yes I get that this is the SRE mantra. But what Pulumi describes here is basically an error budget by a different name. You have just set a lower threshold (ie 'budget') for acceptable errors. Just because a team has an error budget doesn't mean that they don't read the errors until they go over. They are reading and fixing errors to stay within the budget.
- User23 4y agoThis is a great example of the wisdom of not overthinking it. I independently rediscovered this trick early in my career using grep, sort and uniq to get the best up front return. And then later a more sophisticated version with a system that logged to SQLServer. Implementing it as a dev policy is a novelty for me though. I heartily approve.
- beachy 4y agoI guess my age is showing but why would you not read every error? Under what circumstances is it acceptable to ever use a metric like raw number of e.g 500 errors, if you don't know the breakdown of them, whether you're dealing with 50 different errors vs the same error happening 50 times?
- eru 4y agoAt Google, we couldn't have read every single error that eg our Google Photos servers produce, even if we wanted to. At that scale, you get some errors even purely from bits flipping randomly in memory due to cosmic rays. (The latter is speculation. I don't remember how many of those we got, or whether we used enough error correcting memory. It's been a while.)
- mteigers 4y agoDitto, my managed service at Google threw close to 3k 500s / second when I was still there. Anything from cosmic rays, faulty hardware bit flipping, hard drive failures. We did, however, aggregate and group similar 500s and those did get looked at, but no way could we have looked at all errors. The other thing, is that with resilient infrastructure, who cares about an occasional 500. Back off and retry. No harm done.
- willcipriano 4y ago3k? What percentage of requests is that?
- samus 4y agoUser experience might indeed not be influenced by these errors, but errors of a less stochastic nature will impact it. The former obscure visibility of the latter, and that's probably the point of TA.
- MichaelZuo 4y agoIt's likely some fraction of those errors would have had some tangible impact on user experience.
- nonethewiser 4y agoThis is something that probably sounds painfully obvious to someone inexperienced, although they can’t appreciate why it isn’t so obvious. Perhaps this is simply an example of “do things that dont scale” while you can,
- netfortius 4y agoCart before the horses? Why would one question error reading merits, if having a very good understanding of how they're produced, i.e. participate in their configuration and enablement, to begin with?
- arp242 4y agoOne thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean. An error is either: 1. A real bug someone ran in to. 2. Not a bug and just a spurious "error", which should be a log message, some statistic somewhere, or just disabled outright. Errors should only be errors – an uncaught exception, explicit error log, stuff like that. Everything else should be something else. What I've found – and this certainly won't hold true for all cases, but it does seem to be the case often – is that half or more all in the second category: it's just stuff that shouldn't be logged as an error. This is usually trivial to fix. This still leaves you with 5,000 errors/day, but typically ~10 bugs will cause ~4,950 of them. This takes more time to fix, depending on the bug, but usually also takes less time than "5,000 errors every day!" makes it sound. The last ~50 errors are usually the hardest, especially if it's frontend JavaScript stuff. I "read every error", but I have a few that pop up regularly that I just can't reproduce and don't know how to fix. I think it may be extensions people use or something, but idk. Since this post talks about APIs I guess that's not really applicable to them.
- skipkey 4y ago> One thing that I've observed is that the more errors there are, the less likely people will look at them It can be even worse than that. At one point I worked on an app that generated north of 500 lines in the PHP error logs per minute per server. In the course of implementing a feature I cleaned up a few of the low hanging warnings near the code I was modifying. This resulted in about a 3 percent reduction in the log file velocity. When we deployed to prod, everyone panicked, because log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health. That was the one and only time in my career I was reprimanded for cleaning up warnings.
- arp242 4y ago> log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health I don't want to bash on PHP like it's 2002, but ... that's probably one of the most PHP things I ever heard. I worked somewhere where people were against raising error_reporting because someone was in the habit of writing "flase" instead of "false" – which worked because PHP – and it wasn't deemed worth fixing :-/ Old PHP days were wild.
- bdcravens 4y agoI often find myself working on systems where there's a high volume of traffic/processing to developer ratio. Sometimes I'm the only developer. As such, when looking at errors coming in, it's very tempting to take a step back and come up with a methodical triage and error processing framework. However, I find that's a quick path to analysis paralysis. Instead, I'll just start with the most recent error, work through it and push a fix if applicable (whether that's five minutes or five hours), and then move on to the next one. This lets me very quickly add quality back into the system, and very often I find myself resolving other seemingly unrelated issues in the process.
- klodolph 4y ago> Error budgets and the SRE model are haute couture. The error budget does not mean that you don't read the errors. It is just a way to adjust the amount of time you spend fixing bugs versus pushing new features. Also, as mentioned... > Over the next six months, the team iterates, listens to customers, and delivers a ton of value. As a result, traffic levels grow to a rate of 10,000,000 requests per year. The people who advocate looking at errors in aggregate may have much, much busier services.
- doctor_eval 4y agoI was about to write that some kinds of advice are very good in some scenarios and very bad in others. The arguments always seem to skirt that middle ground where good advice goes bad (or vice versa) 10MM requests per year is 20 requests/minute. Most people will probably be working on services at this scale, or less. So the advice is good, but not global, and the qualification is important.
- jacurtis 4y agoI was confused when I saw them mocking error budgets at the end. It shows that they don't understand what error budgets are. What they describe themselves doing is basically an error budget by a different name. We have an error budget at work and we still read essentially every error. There is a long tail of errors that might show up once or twice a week or month and I'll admit that I might have missed those, but within any practical sense, my team is still aware of any error that is contributing against our error budget. We resolve errors too, even while under budgetm because if we didn't then we would go over our error budget since new errors pop up all the time, so you need to resolve existing errors because new ones are introduced constantly. Pulumi seems to think that companies with Error Budgets just wait until they are out of compliance to fix errors and then must only resolve enough to get back under budget.
- fsckboy 4y agook, but don't start looking at runtime errors till you've fixed/eliminated every warning during your build
- fwlr 4y agoRelated advice for when there’s too many errors to read: remember that the original use of triage (in hospitals) was to quickly categorize cases so you could prioritize ones to work on first. It still rapidly terminates, and almost immediately results in individual doctors or teams of doctors working on individual cases. Your triage system will aggregate errors to determine what’s important, sure, but it should quickly tell you what to work on, and (most crucially) those tasks it returns should be disaggregated individual errors to work on. The advantage of software engineering over medical treatment is that for the SRE, solving one case also solves all cases of that type, whereas the doctor has to administer the same treatment to each individual case. But don’t let that scaling trick you into tolerating a triage system that results in tasks like “re-architect this service”.
- sjducb 4y agoIf you're small then I agree. If you've got 100,000 concurrent users then this isn't going to work. 1,000,000 actions per minute means that all of the actions need a 99.9999% success rate to generate 1 error per minute. Lots of 9's of reliability is hard. Also. At that kind of scale you start observing lots of non-deterministic errors such as rare race conditions. As you drill into these rare race conditions you realise that they're affecting 0.001% of customers and you'd have to re-architect your system to eliminate them. You hit a point where it's not worth fixing those errors, so you build an error aggregating system to help ignore them.
- smashed 4y agoOf course any reasonable engineer is going to aggregate similar errors somehow? You don't need to read every single connection reset by peer or similar network errors..
- jacurtis 4y agoI agree with you, but the article specifically mocks error aggregators and says that there is more value in relishing each error. This is counter-intuitive, but seems to be what the article is suggesting.
- samus 4y agoThe authors essentially concur with this view. But comparatively few services have both such high traffic and reliability levels. For everything below, people reach for aggregation way too early.
- rurban 4y agoWith compilers we do have -Werror. This is a godsend, because it forbids crawling up junk like this. Runtime errors are even worse, for this we do have a CI with a 90% test coverage, or better a model checker like cbmc with 100% coverage. Every single error counts. Esp. on hard Realtime, where we reboot on a single error
- jchw 4y agoCounterpoint: my team at Google did desire to read every single error, but still had SLOs and error budgets. In fact, there was a tool specifically for finding and aggregating unique errors so that they could be made into issues. Granted, in practice, that doesn't mean we fixed every single error, but I think that there's no reason to assume "read every single error" conflicts in any way with the typical SRE practices. I'd argue you actually would be doing yourself a disservice to throw away all of the SRE wisdom over this; measuring things quantitatively is definitely not a waste of time.
- lanstin 4y agoThis sounds good, but I don’t understand how it meshes with building reliable systems on top of fallible components. I recently deployed a new service that has three clusters inside it; customer queries hit a proxy layer that queries all three and returns the first non-error reply to the customer. In a typical day, I will have tens of thousands of errors but customers see none. My first deployed code logged the internal errors as errors, but then the SRE started in on a high number of failures, so I switched to counting the internal errors but not reporting as errors, saving that for customer errors.
- brianr 4y agoReading every error is a great idea, but you don't need to read them all yourself - let the computer read them for you. This is what error aggregators (e.g. Rollbar) do. Now you've got 10-1000x leverage on your time because you're reading unique errors instead of every occurrence, and you've got more data to debug with because you've got data about the aggregates ("when did it start? what request params are common? what servers is it on?") in addition to data about the individual occurrences. You can still commit to solving them all ("Rollbar Zero"). It's just a whole lot easier.
- blackflame7000 4y ago[dead]
- jake_morrison 4y agoA customer needed help with the size of their DataDog bill. A big part of that was log messages. They had poor logging discipline, so they started by filtering out all logs that were INFO or lower priority. But then the developers couldn't see what was happening in production, and they started logging messages at WARNING and ERROR in production. That made the output of unit tests in CI light up like a Christmas tree, and the output was unusable. Good times. The solution was to carefully audit all log messages, reducing the number of messages, making sure they were at the right level, and replacing log messages with trace attributes. We also implemented trace sampling to avoid paying for redundant success messages. That reduced the bill by 80%, saving tens of thousands of dollars a month.
- yashap 4y agoIMO - log every 5xx (4xxs too), ensure your logs/error messages are groupable by message (e.g. keep variables like ids in a metadata object separate from the message), and keep a close eye on errors grouped by message and HTTP status code. Then fix any 5xx error that’s too common - IMO, for most cases, this means more than a handful of the same 5xx error most days. And of course, don’t just rely on ppl happening to look at dashboards, alert/page on significant 5xx volume. But there’s no need to send every single error to Slack individually, just keep an eye on counts by error message.