6 ms·
Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside th
by caiobegotti 6y ago
Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think for ourselves?
- mc32 6y agoDon’t bite or embarrass the hand that provides you make-work... Seriously they probably tested it and it worked in theory, just not in practice and now they fix it for reals.
- acruns 6y agoI guess if their DR firedrill assumed their failover router hosting the status page could never go down it would pass, but come on IBM.
- detaro 6y agoSeems a common mistake. If I remember right, AWS stumbled over their status page depending on S3 a few years back
- mbreese 6y agoI view this as growing pains that everyone has to learn the hard way. The bigger question is will they learn the lesson and how will they fix this for the next time? (Because there is always a next time)
- fogetti 6y agoWhile that is a solid advice, still it raises the question: is there such organization which corrects itself by firing the incompetent and promoting the competent instead (the "toldyouso" guy in this case)? Or we just simply accept and making it the norm that even the lowest level of organizational governance is corrupt? I am serious about this, because how people perceive their own rights, their own roles, their own status, their own influence and their organization's wrongdoing will influence the attitude in the long run against each and every organization in society in my opinion. I know that I was blowing the question out of proportion, but it bugged me to ask anyway.
- Avicebron 6y agoI particularly like The Gervais Principle as an explanation of why incompetents are frequently the ones making corporate governance decisions. It's satire, but I think there are elements of truth in it. https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/ https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
- sky_rw 6y agoThese are the same people who recently published a white paper on how they guarantee zero downtime: https://cloud.ibm.com/docs/overview?topic=overview-zero-downtime https://cloud.ibm.com/docs/overview?topic=overview-zero-down... The idea that they could even get to this point probably seemed unfathomable. It does to me.
- dexwiz 6y agoI built a status page for a top cloud provider, and this was question number one from SREs.
- bigiain 6y agoI built a bunch of CloudWatch monitoring for an AWS stack, and duplicated critical monitoring using a 3rd party monitoring service as well. So, because the universe hates me, the 3rd party service migrated their hosting into AWS ~18 months later... :sigh:
- isclever 6y agoIt happens a lot, when you have so much infrastructure and redundancy you think it is too big to fail. Then you lose S3 in US-East1 and break everything. https://www.theregister.com/2017/03/01/aws_s3_outage/ https://www.theregister.com/2017/03/01/aws_s3_outage/
- jedberg 6y agoI would mirror the attitude of the person who said no originally. If they are receptive to feedback and clearly want to do better, I would be kind and explain why I had suggested it not be there in the first place and cite this as an example. If they were being adamant or denying it was their fault, I'd probably be really quiet and just make subtle remarks about how it would have been better if they listened.
- whyleyc 6y agoTotally unrelated Jeremy, but did you know the SSL cert on https://minops.com/ https://minops.com/ has expired? (Was interested to see what you were up to these days, which is how I stumbled on it).
- jedberg 6y agoLol yes. It's intentional. It stops spam bots from trying to sign up, since we aren't open for signups yet. But don't worry, you're not the first to mention it. I suppose I should just fix it and deal with the spam like normal. I liked the unintended effect of cutting down on spam. I guess a lot of spam bots are written on top of standard libraries that reject bad certs. :) Also, this was ironically a great way to publicly call someone out for a seemingly bad decision without being cruel about it, so props to you!
- arkitaip 6y agoThis has to be the most unconventional anti-spam technique I've ever heard about.
- manquer 6y agoIf such people raised concerns and they had been overridden they way you describe they would sadly have left long ago. It is not that companies become consciously malicious or are incompetent to start with, it becomes a vicious cycle, as more and more poor management and engineering talent join, the good ones leave, and the cycle continues. Acquisitions and merge stave off the slow slide into irrelevance for a while, till the best of the new guys leave too. Systemic cultural changes is very very hard to achieve in large organizations.
- snowwrestler 6y agoBest thing to do now is to point DNS at the backup status page they discreetly set up on a free-tier EC2 server back when they got ignored...
- vsareto 6y agoHonestly it's small compared to everything else. I'd rather leave than do told-ya-so though and put the story in the exit interview or reason for leaving.
- all_blue_chucks 6y agoNever humiliate a coworker in public. Instead say "both options were considered but ultimately it was decided to select option B for reason Y."
- caiobegotti 6y agoMy professional experience tells me that the next question will be who decided for B given Y, then you answer it and then you have a target on your SRE back, I'm afraid. Remember that the trickle-down economics works only when the shit hits the fans and what trickles down is not money.
- deleted 6y ago[deleted]
- all_blue_chucks 6y agoIf your company uses post mortems to blame individuals rather than fix processes and tools, you haven't worked in a professional environment.
- FrankPetrilli 6y agoI've worked in these types of organizations before, and it's always counter-productive to making positive changes in the environment. You as an SRE can either make it better, or find an environment that's conducive to positive changes. If you'd like an outside resource to suggest or read up on better postmortem practices, the Google SRE Book has a chapter [0] on postmortem culture. It's an amazing change of pace and a huge stress level improvement for us SREs. [0] https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-...
- stillLicksPenis 6y agoSo then your recommendation, really, is to just jump ship, and turn your back on useless pieces of shit, vindictively isolating them, and never looking back. Don't stand up and fight for what's right. Just turn tail and run, under any circumstances that put you in conflict with leadership and their footguns. Honey, flies, et c...
- skybrian 6y agoIdeally, someone should write a postmortem with a timeline of what happened and recommended fixes. These would then be fixed, and nobody would be blamed. (This is called a "blameless postmortem".) But whether you can get away with that depends on culture.
- epc 6y agoI haven't been at IBM since 2001, but when I was there any suggestion like this would have been beaten down by multiple layers of the big grey cloud for even intimating that such a visible, key piece of IBM marketing material should be on a third party service.
- danjac 6y agoThey would have been told "great, we understand your concerns, please add a card to the tech backlog (or whatever local jargon is at that team) and we'll address it in the next sprint/dev cycle/quarter". That's the right way to acknowledge these complaints while preventing actual progress.