3 ms·
This article comes off as "holy than thou". While maintaining what seems like ignorance in the "haute culture" that they mock. Their primary point of "read eve
by jacurtis 3y ago
This article comes off as "holy than thou". While maintaining what seems like ignorance in the "haute culture" that they mock.
Their primary point of "read every error" is one that almost no one would truly debate (within reason). Most companies do try to read every error as is reasonable. Being aware of an error is different than immediately fixing it (since that can usually be impractical). Where companies tend to differ is in their ability to triage and correct errors. That is the secret sauce that people want to hear.
One of their secondaries points is mocking companies that use error system aggregators. Why is this so bad? They claim it prevents you from reading every error, but I'd beg to differ.
Let's say your system gets 5,000 errors today. It is highly likely that many of those errors are replicate instances of the same error. Out of 5,000 errors you probably have 5 errors that are 800-1,000 instances each, which consumes 4,800 or so of the error instances. Then you probably have a long tail of errors that only show up once or twice a day. So why is an error aggregator bad? Is there any point in reading the same error 1,000 times? Wouldn't it make more sense to read the error once, have the system show you that it shows up 1,000 which is the highest in your system. Then you can spend less time reading more duplicate errors and spend that time fixing them. You can look at a graph and see that the error increases over time, which tells you to prioritize resolution of this error.
If you have 2 hours a day to fix bugs, would you rather fix the bug that causes 2 errors per day or the one that fixes 1,000?
They seem proud of their error burndown chart that shows errors decreasing over time, but this is likely not the result of "reading every error" as much as actually putting time and attention into error resolution. My company for example put a team together at the end of last year who is a cross-functional team who's sole job is fixing prominent errors. We use an error aggregator and our error burndown chart looks the same as Pulumi's. Why? Because we are dedicating resources and attention to fixing it.
Where most companies go wrong is not that they don't read errors. It is that they are so busy pumping out features that they don't dedicate time and resources to fixing things or they don't provide enough time to build less error-prone code before it hits production in the first place.
Lastly, the "haute culture" of error budgets are funny to me. Yes I get that this is the SRE mantra. But what Pulumi describes here is basically an error budget by a different name. You have just set a lower threshold (ie 'budget') for acceptable errors. Just because a team has an error budget doesn't mean that they don't read the errors until they go over. They are reading and fixing errors to stay within the budget.