9 ms·
The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any.
by Anon1096 7mo ago
The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any.
From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. There are certainly problems in Azure, but it's huge and rough edges are to be expected. It mostly marches on. IMO maturity is realizing this and working within the system to improve it rather than trying to lay out all the dirty laundry to an Internet audience that will undoubtedly lap it up and happily cry Microslop.
Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? Azure is still chugging along apparently despite everything being mentioned. People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.
- sabedevops 7mo agoHe might sound like he has a grudge but you sound like you’re personally invested. Shill?
- deleted 7mo ago[deleted]
- kraemahz 7mo agoAWS and Google Cloud are both huge and are significantly better in UX/DX. My only experience with Azure was that it barely worked, provided very little in the way of information about why it didn't. I only have negative impressions of Azure whereas at least GC and AWS I can say my experiences are mixed.
- jiggawatts 7mo ago> People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true. Or… you’ve just normalised the deviation. One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires. After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to wallow with everyone else that has become accustomed to the filth. As an Azure user I can tell you that it’s blindingly obvious even from the outside that the engineering quality is rock bottom. Throwing features over the fence as fast as possible to catch up to AWS was clearly the only priority for over a decade and has resulted in a giant ball of mud that now they can’t change because published APIs and offered products must continue to have support for years. Those rushed decisions have painted Azure into a corner. You may puff your chest out, and even take legitimate pride in building the second largest public cloud in the world, but please don’t fool yourself that the quality of this edifice is anything other than rickety and falling apart at the seams. Remind me: can I use IPv6 safely yet? Does it still break Postgres in other networks? Can azcopy actually move files yet, like every other bulk copy tool ever made by man? Can I upgrade a VM in-place to a new SKU without deleting and recreating it to work around your internal Hyper-V cluster API limitations? Premium SSDv2 disks for boot disks… when? Etc… You may list excuses for these quality gaps, but these kinds of things just weren’t an issue anywhere else I’ve worked as far back as twenty years ago! Heck, I built a natively “all IPv6” VMware ESXi cluster over a decade ago!
- zdragnar 7mo ago> One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires. Eh, I don't think this is exactly as reliable as you'd expect. My previous job had a fairly straight forward code base but had fairly poor reliability for the few customers we had, and the WTF portions usually weren't the ones that caused downtime. On the other hand, I'm currently working on a legacy system with daily WTFs from pretty much everyone, with a greater degree of complexity in a number of places, and yet we get fewer bug reports and at least an order of magnitude if not two more daily users. With all of that said... I don't think I've used any of Microsoft's new software in years and thought to myself "this feels like it was well made."
- jiggawatts 7mo agoThe rapid decay of WTF/day over time applies to both new employees and new customers. > currently working on a legacy system "Legacy" is the magic word here! Those customers are pissed, trust me, but they've long ago given up trying to do anything about it. That's why you don't hear about it. Not because there are no bugs, but because nobody can be bothered to submit bug reports after learning long ago that doing so is futile. I once read a paper claiming that for every major software incident (crash, data loss, outage, etc...) between only one in a thousand to one in ten thousand will be formally reported up to an engineer capable of fixing the issue. I refused to believe that metric until I started collecting crash reports (and other stats) automatically on a legacy system and discovered to my horror that it was crashing multiple times per user per day, and required on average a backup restore once a week or so per user due to data corruption! We got about one support call per 4,500 such incidents.
- zdragnar 7mo agoThe customers aren't pissed, we're doing demos to new departments and lining up customizations and expansion as quickly as we can. We're growing faster than ever within our largest customer. I also didn't say there are no bugs or complaints, I said the system is more stable. But yes, there are fewer bugs and complaints, especially on the critical features. I didn't use the word legacy to mean abandoned, just that it's been around a long time, we're maintaining it while also building newer features in newer tech, as opposed to my previous company which was a green field startup.
- irishcoffee 7mo ago[flagged]
- lokar 7mo agoI think he did kind of point at the lack of seniority in the org, so I'm not sure he was trying to exaggerate with the titles. I'm really struck that they have such Jr people in charge of key systems like that.
- markus_zhang 6mo agoJuniors love to hack out new things and in the mean time they can take the blame if needed, fair trade, won’t you say?
- abtinf 7mo ago> risks to national security …really? Really. Apparently the Secretary of War agrees with him.
- pinkmuffinere 7mo agoTo be fair, it's not like Hegseth is a super high-signal source. Hegseth says lots of stuff, some of which are even true!
- wredcoll 7mo agoThis was such a genuinely weird moment for me when reading the article. "yadda yadda and then also the secretary of defence agreed it was bad" I'm just reading along and going, "yeah that sounds really bad if a secretary level position is being cited... wait a second, isn't that actually the guy who is literally famous for being stupid??" I never expected to be living through a real life version of "the emperor's new clothes", like, how is anyone quoting this guy about anything?
- mpyne 7mo agoIn fairness the SECWAR is hardly a computing expert. But in this case the SECWAR has been properly advised. If anything it's astonishing that a program whereby China-based Microsoft engineers telling U.S.-based Microsoft engineers specific commands to type in ever made it off the proposal page inside Microsoft, accelerated time-to-market or not. It defeats the entire purpose of many of the NIST security controls that demand things like U.S.-cleared personnel for government networks, and Microsoft knew those were a thing because that was the whole point to the "digital escort" (a U.S. person who was supposed to vet the Chinese engineer's technical work despite apparently being not technical enough to have just done it themselves). Some ideas "sell themselves", ideas like these do the opposite.
- jacquesm 7mo ago> If anything it's astonishing that a program whereby China-based Microsoft engineers telling U.S.-based Microsoft engineers specific commands to type in ever made it off the proposal page inside Microsoft, accelerated time-to-market or not. > It defeats the entire purpose of many of the NIST security controls that demand things like U.S.-cleared personnel for government networks, and Microsoft knew those were a thing because that was the whole point to the "digital escort" (a U.S. person who was supposed to vet the Chinese engineer's technical work despite apparently being not technical enough to have just done it themselves). That is beyond bad. Proof of this?
- staticassertion 7mo ago> risks to national security Microsoft is the go to solution for every government agency, FEDRAMP / CMMC environments, etc. > People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true. This I'm more sympathetic to. I really don't think his approach of "here's what a rewrite would look like" was ever going to work and it makes me think that there's another side to this story. Thinking that the solution is a full reset is not necessarily wrong but it's a bit of a red flag.
- kklisura 7mo agoAt no point during the reading I got sense that he's suggesting something radical. Where specifically is he pointing out rewrite? "The practical strategy I suggested was incremental improvement... This strategy goes a long way toward modernizing a running system with minimal disruption and offers gradual, consistent improvements. It uses small, reliable components that can be easily tested separately and solidified before integration into the main platform at scale." [1] [1] https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion-841 https://isolveproblems.substack.com/p/how-microsoft-vaporize...
- staticassertion 7mo ago> The current plans are likely to fail — history has proven that hunch correct — so I began creating new ones to rebuild the Azure node stack from first principles. > A simple cross-platform component model to create portable modules that could be built for both Windows and Linux, and a new message bus communication system spanning the entire node, where agents could freely communicate across guest, host, and SoC boundaries, were the foundational elements of a new node platform Yes, I read that part as well and found it a bit confusing to reconcile with this one. The vibe from my quotes is very much "I had a simple from-scratch solution". They mention then slowly adopting it, but it's very hard to really assess this based on just the perspective of the author. He also was making suggestions about significantly slowing down development and not pursuing major deals, which I think again is not necessarily wrong but was likely to fall on deaf ears.
- bawolff 7mo ago> Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? That struck me too. Maybe i've never worked high enough in an org (im unclear how highly ranked the author of the piece is) but i've never been in an org where going over your boss's boss's boss's boss's head and writing a letter to the board was likely to go well. That said, i could easily believe that both Azure is an absolute mess and that the author of the piece was fired because of how he went about things.
- beoberha 7mo ago[flagged]
- 0______0 7mo agoRedacted to avoid getting doxxed (my original reply showed disdain for the parent comment and agreed with Axel's writing).
- axelriet 7mo agoFormer 1010 Overlake RnD here too :)
- RajT88 7mo agoIt reminded me of this one: https://wtfmitchel.medium.com/how-to-get-fired-from-microsoft-part-1-cc2980d33bfe https://wtfmitchel.medium.com/how-to-get-fired-from-microsof... A lot of similarities, except the medium author was not part of PG but support. He also had recently suffered a brain injury.
- nrds 7mo agoI've worked at both Microsoft and Google in the past 6 years and the notion that msft "Principal" is equivalent to goog L5 is crazy.
- dh2022 7mo agoMeaning Msft Principal is below L5? I got the same feedback from one of my friends who works at Google. She said quality of former MSFT engineers now working at Google was noticeably lower.
- zipy124 7mo agoI mean if you go by pay in the UK a Microsoft principle is equivalent to an L4 at Google if levels.fyi is too be believed....
- nrds 6mo agoI mean imputed prestige within the organization. Being an L5 is nothing; it's the promote-or-fire cutoff at Google AFAIK. But being a Principal is slightly more than nothing; it's two levels above the promote-or-fire cutoff. I mean, _now_, sure, I'd assume Microsoft Principals should be hired around L4 at Google. But that's just due to a temporary inbalance in the decline of legacy organizations. Give it a few years and it will even back out and msft 64 will be in the middle of L5 range like levels.fyi claims.
- compiler-guy 6mo agoL5 hasn't been the promote or fire cutoff at Google for perhaps a decade. L4 is the new L5, mostly because Google would have to pay L5s more, and it has been terrified of personnel costs for years. But even so, an L5 at Google is basically a nobody as far as prestige or convincing other people to adopt your plan goes. Even L6 is basically just an expert across several mostly local teams. L7 is where the prestige gets going.
- rawgabbit 7mo agoI believe the author was referring to this https://www.propublica.org/article/microsoft-digital-escorts-pentagon-defense-department-china-hackers https://www.propublica.org/article/microsoft-digital-escorts.... Microsoft hired Chinese engineers to manage US Department of Defense Azure VMs.
- axelriet 7mo agoThanks. That reference is correct. The point is why those sessions were necessary because there is no reason, a-priori, to do manual touches on production systems, DoD or not.
- com2kid 7mo ago> From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. Before the days of title inflation across the industry, a a Principal at Microsoft was a rare thing. When I was there, the ratio was maybe 1 principal for every 30 developers. Principals were looked up to, had decades of experience, and knew their shit really well. They were the big guns you called in to fix things when the shit really hit the fan, or when no one else could figure out what was going on.
- Anon1096 7mo agoOne of Microsoft's problems is their pay is significantly lower than FAANG and so you very very rarely see people with expertise in the same verticals jump to Azure. I get that "the deal" at Microsoft is lower pressure for lower pay but it really hinders the talent pipeline. There are some good home grown principals and seniors, but even then I think the people I worked with would have done well to jump around and get a stint at another cloud provider to see what it's like. Many of them started as new grads and their whole career was just at Azure. Meanwhile when I was at another company we would get a weekly new hire post with very high pedigree from other FAANGs. And with that we got a lot of industry leading ideas by osmosis that you don't see Azure getting.
- fleshdaddy 6mo agoYeah the deal has also changed. Right as I was leaving the messaging started changing a lot and there was a clear top down “you all need to work harder”. They hired an ex Amazon guy to run my org which really drove the message home. To be fair though I think Microsoft has decided they are fine with rank and file being mediocre. I don’t know how interested they are in competing for top talent except for at the top.
- com2kid 6mo ago> I get that "the deal" at Microsoft is lower pressure for lower pay but it really hinders the talent pipeline. The deal used to be a lower cost of living in a major coastal city, an amazing campus (it is seriously lovely), every engineer had their own office, serious job security, and an unbelievable health care plan. Seattle exploded in price, they moved to open offices, Microsoft started doing mass layoffs, and they gutted the healthcare plan (by the time I left the main plan on offer was a high deductible with a miserable prescription formulary). Hard to attract talent when there is no big differentiator. Of course in the 90s the deal was work there 10 years retire a millionaire. Easy to attract talent when that is the offer ...
- whoamii 7mo agoFor reference, author was a Senior Software Engineer, ie. mid-level engineer.
- markus_zhang 6mo agoHe was from the kernel team, though. I always put their experience x2.
- whoamii 6mo agoUnless you’re still senior after 10+ years. Add to that switching companies and coming back same level…
- markus_zhang 6mo agoI don't disagree with you. I wish there were some good counter points from the Azure team. There was one from the Azure team in who commented on the article, but I feel that comment to be a bit weak.
- axelriet 6mo agoLife works in mysterious ways. Whoever you are, bring it on and prove any of my points wrong.
- whoamii 6mo agoLike the one where 1.5T in value went pfoof due to reasons you mentioned? I will let people judge whether your arguments are most likely, or whether this is bubble syndrome. Hint: there is a large distance between use of smart pointers and market effects.
- axelriet 6mo agoHint: there is a small distance, almost zero, between your product being unreliable and losing your most flagship customer and the trust of the DoD. Factor that there are limits to what I was willing to share in that story, limits that I’ll certainly not cross here. Now, if you think that I’m wrong and that Microsoft found an entirely different way of shooting itself in the foot, I’d love to hear your perspective.
- Hikikomori 7mo agoChugging along? Very clear you're not a customer using Azure.
- axelriet 7mo ago>risks to national security and sending letters to the board, really? Yes, really, and guess what the DoD did on Aug 29, 2025, exactly 234 days after I warned the CEO of potential risks? https://www.propublica.org/article/microsoft-china-defense-department-digital-escorts-investigation-warning https://www.propublica.org/article/microsoft-china-defense-d... It wasn’t specifically about the escort sessions from any particular country, though, but about the list of underlying reasons why direct node access was necessary.
- small_model 7mo agoYes it's easy to critique any large system or organisation, to then go over everyone's head and cry to the CEO and Board is snake like behaviour especially offering you self as the answer to fix it. OP will be marked as a troublemaker and bad team member.
- axelriet 7mo agoMaybe. That would be a dent in the shiny culture of trust Microsoft is proud to run on, though.
- UltraSane 7mo agoThe problem is that what he writes is very plausible and explains a lot about why Azure is so unreliable and insecure. The author didn't mention the shameful way Microsoft leaked a Golden SAML key to Chinese hackers. This event absolutely was a threat to national security.
- dgellow 7mo agoIf your reaction is emblematic of the way people reacted to his points internally that does give more credibility to his side of the story IMHO
- axel479343 6mo agoThe grudge is simple and doesn't detract one thing from a very well articulated blog: you do you job as an engineer of pointing out problems, even proposing solutions, and they fire you for doing exactly the job. It's infuriating enough just from reading it, idk how you can't see any legitimacy on what the guy is complaining. You have your right of free speech to complain about shitty jobs if you want, there's no honor bound to maintain silence here.
- dartharva 6mo agoDo you contest the fact that Microsoft royally fumbled OpenAI out of sheer incapability of providing what's supposed to be its core business despite having all deals in its favor? Because that's the most damning validation against Azure in recent times.