3 ms·

At large bureaucratic organisations there's often political implications of changing the official status, so often it lags behind reality until it cannot be swe

At large bureaucratic organisations there's often political implications of changing the official status, so often it lags behind reality until it cannot be swept under the rug anymore Not saying it's right, just an observation

KingOfCoders 6y ago
As a CTO responsible for an often failing eCommerce website that lost millions when down and which I took over, I fell in the same trap of trying to sweep things under the rug. Until I decided to no longer do that and my life improved considerably.
- malux85 6y ago
  Yeah it's very frustrating - especially if your customers are technical, they are seeing the errors and the status page says everything is fine. I've seen status pages and error counts tied to bonuses, which only caused a giant mess of bad incentive alignment and internal lies, customers are unhappy, developers are unhappy, management are lying to upper management, it's so much easier to focus efforts on real problems and just be honest and improve. Thank goodness I dont work there anymore (cough cough Google)
  - penagwin 6y ago
    Do these companies not have live error reporting and tracing? Like surely github got alerts that things weren't working? Why don't they just hookup their status page and their alerts? Or is it a political/relationship thing, and they want to have a human give out the status page updates? This could have been caught with a cron job and some curl requests :\
    - mjayhn 6y ago
      In all honesty they're typically just banking on people not noticing it and are trying to make it as little of a fuss as possible and get it up before it gets to twitter. The problem is when it's not just a small blip and they haven't addressed it and it goes mainstream and is still down, it just leads to concerns about transparency. Building infra I have to work around all sorts of 3rd party services going out or having blips throughout the day, docker registries, caches, bgp, etc., it's totally an expected part of infra design but not every team has the time or need to build in the resiliency. I see tons of outages that never get reported or IMO aren't reported adequately enough. With that said, I'm no angel, I get all my service down notifications through slack, so when slacks down..
JMTQp8lwXL 6y ago
What's the point in having a status page if it's a political artifact? In that case, it serves zero customer value.
- opportune 6y ago
  It’s still an indicator that “no you’re not crazy, we’re having issues on our end” but not a foolproof one. It’s kind of sad that twitter is usually the best place to confirm an outage as it begins, rather than the software providers themselves. I assume if they actually exposed global availability metrics in most cases it would not look as good as they would want it to
  - paulie_a 6y ago
    Down detector is based on twitter complaints and is pretty damn accurate.
tfolbrecht 6y ago
3 words: Service Level Agreement I've caught a big cloud provider not reporting a degraded service, I assume they knew but politics and $ come in and it's easier to just gaslight everyone. I get it, but my frustration is worth loosing a trailing 9. I think there should be some 3rd party continuously testing APIs. Degraded states are downtime!
tdeck 6y ago
Honestly I think this can be true at any size organization. Small startups often take the approach of "let's hope no one noticed while we try to fix it", it's just that they have fewer users to notice so it's more likely to work.

KingOfCoders 6y ago

As a CTO responsible for an often failing eCommerce website that lost millions when down and which I took over, I fell in the same trap of trying to sweep things under the rug. Until I decided to no longer do that and my life improved considerably.

malux85 6y ago
Yeah it's very frustrating - especially if your customers are technical, they are seeing the errors and the status page says everything is fine. I've seen status pages and error counts tied to bonuses, which only caused a giant mess of bad incentive alignment and internal lies, customers are unhappy, developers are unhappy, management are lying to upper management, it's so much easier to focus efforts on real problems and just be honest and improve. Thank goodness I dont work there anymore (cough cough Google)
- penagwin 6y ago
  Do these companies not have live error reporting and tracing? Like surely github got alerts that things weren't working? Why don't they just hookup their status page and their alerts? Or is it a political/relationship thing, and they want to have a human give out the status page updates? This could have been caught with a cron job and some curl requests :\
  - mjayhn 6y ago
    In all honesty they're typically just banking on people not noticing it and are trying to make it as little of a fuss as possible and get it up before it gets to twitter. The problem is when it's not just a small blip and they haven't addressed it and it goes mainstream and is still down, it just leads to concerns about transparency. Building infra I have to work around all sorts of 3rd party services going out or having blips throughout the day, docker registries, caches, bgp, etc., it's totally an expected part of infra design but not every team has the time or need to build in the resiliency. I see tons of outages that never get reported or IMO aren't reported adequately enough. With that said, I'm no angel, I get all my service down notifications through slack, so when slacks down..

JMTQp8lwXL 6y ago

What's the point in having a status page if it's a political artifact? In that case, it serves zero customer value.

opportune 6y ago
It’s still an indicator that “no you’re not crazy, we’re having issues on our end” but not a foolproof one. It’s kind of sad that twitter is usually the best place to confirm an outage as it begins, rather than the software providers themselves. I assume if they actually exposed global availability metrics in most cases it would not look as good as they would want it to
- paulie_a 6y ago
  Down detector is based on twitter complaints and is pretty damn accurate.

tfolbrecht 6y ago

3 words: Service Level Agreement I've caught a big cloud provider not reporting a degraded service, I assume they knew but politics and $ come in and it's easier to just gaslight everyone. I get it, but my frustration is worth loosing a trailing 9. I think there should be some 3rd party continuously testing APIs. Degraded states are downtime!

tdeck 6y ago

Honestly I think this can be true at any size organization. Small startups often take the approach of "let's hope no one noticed while we try to fix it", it's just that they have fewer users to notice so it's more likely to work.