6 ms·
Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashb
by uji 6y ago
Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not.
Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view.
As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins) so most of the time things are broken, its just that its not visible to external customers through dashboard.
- etaioinshrdlu 6y agoWow. If I were in charge, the team running a service should not be the same team who decides whether a given service is healthy. This is pretty damaging info about the unprofessional way AWS actually appears to be run.
- foota 6y agoI think this is pretty typical, as often outsiders don't have the visibility into the issue to determine whether there's an issue.
- alfalfasprout 6y agoIt's funny that you point to that as the problem. The problem is more AWS' toxic engineering culture that has engineers fearing for their jobs in a way that guides their decision making. It's bad company culture, end of story.
- etaioinshrdlu 6y agoAs a customer I don't really care whether AWS has a toxic internal culture. I care about whether they have operational excellence and a high quality product. This information is showing cracks in operational excellence.
- martamorena9182 6y agoAWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general. You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state. But the matter of fact is simple. If you end up in a team like this, switch and raise complaints afterwards. Nothing stops you from it. There is no "toxic engineering culture" at AWS. The problem is that AWS makes you into an owner and that includes owning your career. That means if you feel something is wrong, YOU are expected to act. No one will do it for you. And there are plenty of mechanism for you to act. This is the greatest benefit of working at Amazon but its also the downfall of people who are not able to own things.
- Aeolun 6y ago> The problem is that AWS makes you into an owner and that includes owning your career. Firing me for correctly telling customers that their services are down is not my idea of making me an owner.
- Frost1x 6y agoYou're the owner of aspects like responsibility and risk but not the owner of aspects related to financial growth (I mean, your stock options are, but that's about it).
- martamorena9182 6y agoDoing what you think is right, is not necessarily the right thing to do. This is why there is also "Disagree and Commit". There are many facets to this and I am 100% sure that you did not get fired for >correctly< telling customers... You could potentially get fired for incorrectly telling them though, if the issue was severe enough.
- Ensorceled 6y agoThat sounds toxic.
- dilyevsky 6y agoGuess what - most cloud providers are like that. My personal experience is with GCP where stuff can be majorly on fire and no status update for hours. Cloud SLOs are lies like a lot of other things there
- o-__-o 6y agoMy company will update their status but puts the most vague responses up. Reason is because we don’t want to appear inept when we crash the website. For example, because we ran out of disk space Our competitors would have a field day with that
- ta20200710 6y agoThe ec2 or s3 dashboards showing red literally requires approval from ajassy himself irrc The status page is entirely manually updated.
- ipsocannibal 6y agoFlipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.
- reaperducer 6y agoAll I see in your response is, "We don't want to tell the truth because it might cost us money." Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.
- deleted 6y ago[deleted]
- Frost1x 6y agoOh, what a pipedream. If only capitalism worked how it was described in textbooks. It turns out there are much easier lower cost optimizations businesses can perform based on managing perception rather than worrying about pesky concepts like utility.
- woahAcademia 6y agoI never considered Amazon capitalistic given their exploitation of the USPS. I considered them this private company subsidized by taxes.
- AmericanChopper 6y ago
- totaldude87 6y agowow, so much to their "leadership principles" , the first one being "customer obsession" and "earning trust", from what I see, this doesn't accomplish either :|
- o-__-o 6y agoI’ve got another good FAANG principal joke: “Don’t be evil” buys doubleclick
- deleted 6y ago[deleted]
- martamorena9182 6y agoIf you don't get the non-existant relation between Sev2s and the public dashboard, then that says a lot about your qualification to post about AWS on HN.