6 ms·
Microsoft blames outage on small staff, automation failures
- gardenhedge 3y agoDidn't they just fire a load of people?
- mnd999 3y agoI guess it’s double shifts for those that are left. Overwork and tiredness never caused any problems whatsoever, right?
- Cthulhu_ 3y agoFrom an armchair point of view, Microsoft seems like the kind of company that will have armies of middle managers whose job is to be seen with shelves of paper and a cup of coffee moving through offices, or other kinds of bullshit jobs.
- Dudester230602 3y agoThat probably best describes Google these days to be honest. All that alleged engineering might yet not much new stuff to show up for it.
- Sparkyte 3y agoYou can't eliminate jobs to make up loss without risk. Fire middle management before firing engineers.
- highwaylights 3y ago"It is difficult to get a man to understand something, when his salary depends upon his not understanding it" Middle managers aren't going to give up their salaries when there are perfectly good underlings to sacrifice first, especially when they can just tell the Chat GPT to do the codes like they read in that ebook they bought last night.
- Sparkyte 3y agoIt'll never be that simple. Middle managers want to imagine and dream in a place where AI is the bottom tier of their worker category, but in the end of the day it makes them bottom tier. A way I've always explained it to people is that ChatGPT is based off our knowledge and if knowledge is never improved by a human constantly updating ChatGPT it will never improve it will just make up shit to fill in the blanks and it will not be free of error. AI can loose coherency on it's data if the data it is training on is full of errors. Like data death due to compressing what is compressed over and over again.
- baz00 3y agoSo the root cause is incompetence. They didn't have enough people and they didn't have a fully tested recovery plan, relying on assumptions and winging it.
- badcppdev 3y agoImportant to note that the root cause is incompetence at the upper management level.
- AnthonyMouse 3y ago> Microsoft also had trouble understanding why its storage infrastructure didn't come back online. > Storage hardware damaged by the data hall temperatures "required extensive troubleshooting" but Microsoft's diagnostic tools could not find relevant data because the storage servers were down. Should've self-hosted it instead of trusting some cloud vendor.
- nixgeek 3y agoWhat do you do when you are the cloud vendor?
- nottheengineer 3y agoYou do better than this so your customers don't start hosting themselves.
- andyjohnson0 3y agoObligatory XKCD: https://xkcd.com/908/ https://xkcd.com/908/
- vb-8448 3y agoCloud vendors should focus more on "low/mid level" stuff(hardware and basic virtualization) and less on all their abstraction that should be left to end-users or third party software vendors. Unfortunately, this will never happen.
- vmfunction 3y agoThis is what co-location do. Some hosting company still does that. So look for those, then you just bring your own hardware appliances. Then you just using their facilities, uninterrupted power supplies.
- vb-8448 3y agoI don't want to mess with my hardware. Anyway, with "abstractions" I mean all services like AWS App Runner that are build on top of foundational services (ec2, ebs, s3, vpc) that drains resources and money at the expense of the low level stuff.
- WJW 3y agoHN headline is quite different than the actual article, which is "Microsoft admits slim staff and broken automation contributed to Azure outage". The headline as submitted is quite focused on the manager vs employee conflict that is so in vogue on HN at the moment, while in the actual headline (and the rest of the article) Microsoft acknowledges they've put too few people on that shift and have already taken mitigations. Mitigations both to increase the team size and improve the automation that supports them btw.
- greggsy 3y agoI read it in the context of management stating that they’re understaffed. Different worldviews and perspectives I suppose.
- Torkel 3y agoThe word "small" in the HN headline ("Microsoft blames outage on small staff [...]"). I am not native English, but the image in my head was of the staffers being physically small. A bit curious how a native speaker reads it?
- vb-8448 3y agoSo the cost reduction that all these hyperscalers pledge doesn't come from huge volumes, but from saving on staff? Just a rant ...
- benterix 3y agoMaybe a rant, but there is some truth in it.
- azangru 3y ago> Microsoft blames... Is this the purpose of incident analyses? Blame?
- Cthulhu_ 3y agoJust scanning the actual report / postmortem, they don't seem to "blame" the staff at all - that's The Register's journalistic freedom / interpretation at play. The only thing mentioned about staff size is the following: > Due to the size of the datacenter campus, the staffing of the team at night was insufficient to restart the chillers in a timely manner. We have temporarily increased the team size from three to seven, until the underlying issues are better understood and appropriate mitigations can be put in place. Status report is at https://azure.status.microsoft/en-us/status/history/ https://azure.status.microsoft/en-us/status/history/, the tracking code is VVTQ-J98
- mdriley 3y agoThe report says the cooling issue caused "a loss of service availability for a subset of [one] Availability Zone". How did a single-AZ failure cause outages for two dozen services? Why did a single-AZ failure mean "approximately half of Cosmos DB clusters in the Australia East region were either down or heavily degraded" and require those clusters to do a cross-region failover?
- aaron695 3y ago[dead]
- Stranger43 3y agoI suspect the big dark secret of the public cloud is that some regions/AZ's are not all that big and that there is no secret tech that's the big players have that the smaller regional players dont when it comes to the actual physical layer.