7 ms·
As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics i
by phyzome 3y ago
As an engineer on call, I have been in this conversation so many times:
"Hey, should we go red?"
"I don't know, are we sure it's an outage, or just a metrics issue?"
"How many users are affected again?"
"I can check, but I'm trying to read stack traces right now."
"Look, can we just report the issue?"
"Not sure which services to list in the outage"
...and so on. Basically, putting anything up on the status page is a conversation, and the conversation consumes engineer time and attention, and that's more time before the incident is resolved. You have to balance communication and actually fixing the damn thing, and it's not always clear what the right balance is.
If you have enough people, you can have a Technical Incident Manager handle the comms and you can throw additional engineers at the communications side of it, but that's not always possible. (Some systems are niche, underdocumented, underinstrumented, etc.)
My personal preference? Throw up a big vague "we're investigating a possible problem" at the first sign of trouble, and then fill in details (or retract it) at leisure. But none of the companies I've worked at like that idea, so... [shrug]
- virtue3 3y agoI think your bit at the end is the most important. ANY communication is better than no communication "everything is fine, it must be you" is the worst feeling in these cases. Especially if your business is reliant on said service and you can't figure out why you are borked (eg the github ones).
- taneq 3y agoOnce an ISP support person insisted that I drive down to the shop and buy a phone handset so I could confirm presence of a dial tone on a line that my vdsl modem had line sync on before they’d tell me their upstream provider had an outage. I was… unimpressed.
- throwaway167 3y agoYour point highlights thinking about what's being designed. everything is fine is different from nothing has been reported. A green is misleading, there should be no green as green is unknown, there should be nothing with a note that there's nothing, and that's not the same as a green light.
- petesergeant 3y ago> ANY communication is better than no communication Better for the consumer, although not necessarily better for the provider if they have an SLA.
- smsm42 3y agoIMHO, any significant growth in 500s (that's what I was getting during the outage) warrants mention on status page. I've seen a lot of stuff, so if I see an acknowledged outage, I'll just wait for people to do their jobs. Stuff happens. If I see unacknowledged one, I get worried that people who need to know don't and that undermines my confidence in the whole setup. I'd never complain if status page says maybe there's a problem but I don't see one. I will complain in the opposite case.
- TeeWEE 3y agoConnect your status page to actual metrics and decide a treshold for downtime. Boom you’re done.
- lrem 3y agoDoes anyone serious do this? That’s an honest question, from a pretty experienced SRE.
- darkwater 3y agoIn a world of unicorns and rainbows, absolutely. In the real world, it's as you probably already know: it's not that easy in a complex enough system. Quick counter-example for GP: what if the 500 spike is due to a spike in malformed requests from a single (maybe malicious) user?
- laeri 3y agoA malformed request should not lead to a 500, they should be handled and validated.
- jon_adler 3y agoTrue, however it also doesn’t impact other users and doesn’t justify reporting an incident on the status page.
- darkwater 3y agoWell, in the real world it might. It should trigger a bug creation and a fix to the code, but not an incident. Now all of a sudden to decide this you need more complex and/or specific queries in your monitoring system (or a good ML-based alert system), so complexity is already going up.
- laeri 3y agoQuery input validation is nearly a solved problem. If you don't I would argue this is an incident if in this case 500's are returned.
- PeterStuer 3y agoAnd that is before 'going red' has ties to performance metrics with SLA impacts ...
- DANmode 3y agoWhich then means not going yellow or red technically constitutes fraud.
- PeterStuer 3y agoNot necessarily. The situation can be genuinely unclear to the point where it is a judgement call, and then it becomes a matter of how to weigh the consequences.
- DANmode 3y agoIf you're asking how many users are affected, and your service is listed as green...
- PeterStuer 3y agoWhat if the answer is 0.00001%?
- saagarjha 3y agoStill seems like a yellow to me.
- Gareth321 3y agoThis is exactly why those status pages are almost always a lie. Either they need to be fully automated without some middle manager hemming and hawing, or they shouldn’t be there at all. From a customer’s perspective, I’ve been burned so many times on those status pages that I ignore them completely. I just assume they’re a lie. So I’ll contact support straight away - the very thing these status pages were intended to mitigate.
- asah 3y agomeh - no status page is perfectly in sync with reality, even if it's updated automatically. There's always lag and IRL, there's often partial outages. Therefore, one should treat status pages conservatively as "definitely an outage" rather than "maybe an outage."
- AndrewKemendo 3y agoThe simple fix is to have a “last update: date time” Or you can build a team to automate everything and force everyone and everything into a rigid update frequency which becomes a metric that applies to everyone and becomes the bane of the existence of your whole engineering organization