21 ms·

Decisions that eroded trust in Azure – by a former Azure Core engineer

• axelriet 7mo ago
  A former Azure Core engineer’s 6-part account of the technical and leadership decisions that eroded trust in Azure.
  • AceJohnny2 7mo ago
    I downvoted this comment for sounding like a summarizing LLM, not adding anything substantial beyond the title of the post, before realizing you were the poster and author.
    • axelriet 7mo ago
      I didn’t know that “subtitle” would appear as first comment.
      • dijksterhuis 7mo ago
        huh, i didn't realise that's what that does either
  • ninininino 7mo ago
    What's your assessment of AWS and GCP? Do you think it's likely they suffer from some of the same issues (eg the manual access of what should be highly secure, private systems, the instability, the lack of security)?
    • rybosome 7mo ago
      As a former GCP engineer, no, the systems are not generally unstable or insecure. There is definitely manual access of data - it requires what was termed “break glass” similar to the JIT mechanism described by the author. However, it wasn’t quite so loose; there were eventually a lot of restrictions on who could approve what, what access you got after approval, and how that was audited. It was difficult to get into the highest sensitivity data; humans reviewed your request and would reject it without a clear reason. And you could be 100% sure humans would review your session afterwards to look for bad behavior. I once had to compile a large list of IP addresses that accessed a particular piece of data to fulfill a court order. It took me days of effort to get and maintain the elevated access necessary to do this. I have a lot of respect for GCP as an engineering artifact, but a significantly less rosy opinion of GCP as an organization and bureaucratic entity. The amount of wasted effort expended on engaging with and navigating the bureaucracy is truly mind-boggling, and is the reason why a tiny feature that took a day to code could take months to release.
      • Terretta 7mo ago
        Insecure is a curious word as it entangles with what is or isn't known, more than informs about design. A different way to put it is GCP architecture has made different tradeoffs. For example favoring operability over confidentiality*, or scalability over integrity. This makes sense from its mono-tenant engineering origins. Those were the right calls. Google exported SRE not SecEng. Frankly, for most cloud customers, it's what they need. --- * Take this break glass process. It arguably shouldn't be possible. If clients need their CSP to be "NSL proof", unable to leak corporate info responding to a national security letter (or any less obligatory rationale) without the corporation knowing, GCP is not their cloud. CSPs mostly consider it more difficult than it's worth to design a cloud offering that can be proven unable to provide a client's data. On the contrary, customers yell if CSP can't restore lost data, like Apple users yell if Apple can't restore iCloud. iCloud Advanced Security is what happens when you build clients the choice -- witness the warnings. Support drives design choices, not security.
  • jacquesm 7mo ago
    Why do you speak about yourself in the third person? Also, after this: https://news.ycombinator.com/item?id=20341022 https://news.ycombinator.com/item?id=20341022 You continued to work at Microsoft and now there is this takedown? I'm no friend of MS (to put it very mildly) but it seems to me your story is a bit inconsistent as well as the 7 year break between postings.
    • netruk44 7mo ago
      > Why do you speak about yourself in the third person? When you submit a link to HN, there is an entry field for text in addition to the url. It does not really describe what the text is used for. For links, the content of that field is simply added as the first comment. Someone who is unfamiliar with the submission process may assume this field should describe what they are submitting, and not format it like a comment. Then that text gets posted as the first comment and tons of people downvote it, jumping to the conclusion that the weird summary comment is from an AI, and not the submitter describing their own submission. (I also assumed these comments were AI until someone else pointed this out)
      • dnw 7mo ago
        AH! Thanks, that's useful context!
      • axelriet 7mo ago
        Could not have said it better myself. Thanks.
    • axelriet 7mo ago
      The comment comes from the input field on the post form. Not clear it would show up as a comment. The old thread you refer to had little to do with Microsoft per se. Let me known if I can help with the inconsistencies you mention?
• pRusya 7mo ago
  It's a nice read. Thank you for sharing this. > Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls. This is what people should know when seeing massive layoffs due to AI.
  • maxwg 7mo ago
    I honestly thought this was one of the weaker points of the article. The OpenAI deal almost certainly related purely to GPU capacity, which had little to do with the article. The layoffs would have happened regardless. IMO - churn, and generalization is the root cause. Engineers are thrown on projects for a year with little prior experience, leave others to pickup the pieces, etc. There's no longer a sense of ownership, and I'm sure the recent wave of layoffs isn't helping with this.
    • axelriet 7mo ago
      GPUs is something that can be fixed simply by throwing money at it.
    • hrmtst93837 7mo ago
      [flagged]
    • hrmtst93837 7mo ago
      [flagged]
    • hrmtst93837 7mo ago
      [flagged]
    • deleted 7mo ago
      [deleted]
    • deleted 7mo ago
      [deleted]
    • deleted 7mo ago
      [deleted]
• Bjartr 7mo ago
  What a fascinating view into how the sausage is made
• nope1000 7mo ago
  > The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical. > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected. That is quite scary
  • deleted 7mo ago
    [deleted]
  • xorcist 7mo ago
    Scary is the understatement of the day. I can't imagine the environment where someone think that architecture is a good idea.
    • axelriet 7mo ago
      And yet, there we are.
  • jldugger 7mo ago
    Like, what did the OP expect?
  • dh2022 7mo ago
    This is well documented: https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=windows https://learn.microsoft.com/en-us/azure/virtual-machines/ins... Why would an Azure customer need to query this service at all? I was not aware this service even exists- because I never needed anything like it. AFAI can tell, this service tells services running on the VM what SKU the VM is. But how is this useful to the service? Any Azure users could tell how they use IMDS? Thanks!
    • lokar 7mo ago
      To have a new vm configure itself at boot
    • 7mo ago
• schlauerfox 7mo ago
  "For fiscal 2025, Microsoft CEO Satya Nadella earned total pay of $96.5 million, up 22% from a year earlier." -CNBC.com and "I also see I have 2 instances of Outlook, and neither of those are working." -Artemis II astronaut
  • tantalor 7mo ago
    > 2 instances of Outlook That's 2 too many.
    • bigbuppo 7mo ago
      They should have used the third outlook they didn't know about... Outlook, Outlook (new), and the well-hidden Outlook (classic) that actually works.
      • cyanydeez 7mo ago
        That outlook was part of the ablative outlook armor thats suppose to burn off on reentry
        • cebert 7mo ago
          Do you have a source for that? I don’t see what impact consumer email software would have with the composition of the heat shield.
          • anonymars 7mo ago
            I believe this one would fall under "incongruity theory" https://en.wikipedia.org/wiki/Theories_of_humor#Incongruity_theory https://en.wikipedia.org/wiki/Theories_of_humor#Incongruity_...
          • john_strinlai 7mo ago
            its a joke, no sources required
          • 7mo ago
• gnabgib 7mo ago
  Title: How Microsoft Vaporized a Trillion Dollars
  • axelriet 7mo ago
    As an investor, this is exact how I feel. Everything was skyrocketing until OpenAI “diversified” mid-2025. The company’s market value has dropped more than 1 trillion since late October 1025, so the title is factual. You can rightfully argue and be skeptical about the link I make, but not about the numbers :)
    • axelriet 7mo ago
      OK, *2025
• vintagedave 7mo ago
  What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements. > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary. Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern? More, why is this public not a court case for wrongful termination? Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
  • bigbuppo 7mo ago
    The CEO is accountable to the board. If they are derelict in their obligations to the company, that's where you need to raise a stink so they can fix it.
    • ohyoutravel 7mo ago
      Well, yeah, that’s what a board does, but I think the issue is whether it is customary to go to the board directly in this situation. The answer is a resounding NO. Very odd, but cool idea and approach.
      • ryandrake 7mo ago
        Maybe naive, but why not? If it's a serious enough issue, and you're not getting anywhere through your management chain all the way up to the CEO, why is it novel to contact the people the CEO reports to? They're not royalty, they're other human beings who also eat, piss and fart like everyone else.
        • refulgentis 7mo ago
          Before 6 years of Google I’d co-sign what you said, but it never ever plays out that way. The law of the jungle is an iron law, make people around you feel bad, be a tattletale, and you’re choosing to be ostracized. That said yr interlocutor disturbs me a bit because yes, they certainly will make it out to be a mental health episode. But the implicit deal there is “STFU. You can even take paid health leave.” It’s not healthy either. BigCo is insane I’ll never work for one again without outrageous comp. You’d be stunned by even the simplest story. Ex. a year in some crazy shit was going down and my manager asked for my thoughts on a topic, I was honest and basically said “I don’t think it’s a good idea, but in my experience, raising issues involving people only raises more issues.” He swore up and down it wouldn’t be a problem, eventually made a deal I could email it to him privately. Next 1:1 with my area lead was horrible, them seeing red, hearing a mistranslated version of what I said, and I had 0 warning.
• brcmthrowaway 7mo ago
  What an epic takedown. Microsoft should have promoted this guy instead of laying him off. Did Microsoft really lose OpenAI as a customer?
  • axelriet 7mo ago
    The answer to your question is in the public releases. MS went from primary partner (under ROFR) to one of the options. They retain IP rights and API hosting, although in recent weeks we learned that OpenAI was planning a workaround with AWS and Microsoft said they might sue them for that. So the happy marriage is over, it’s more like a custody battle now: https://www.reuters.com/technology/microsoft-weighs-legal-action-over-50-billion-amazon-openai-cloud-deal-ft-2026-03-18/ https://www.reuters.com/technology/microsoft-weighs-legal-ac...
• andrewstuart 7mo ago
  Any complex system - and these cloud systems must be immensely complex - accumulate cruft and bloat and bugs until the entire thing starts to look like an old hotel that hasn’t been renovated in 30 years.
  • lll-o-lll 7mo ago
    It’s not inevitable. Absolutely this is true without significant effort, but if you’ve been around the traps for long enough (in enough organisations), you get to see that the level of quality can vary widely. Avoiding the mud-pit does require a whole org commitment, starting from senior leadership. This story is more interesting, in my opinion, in how quickly things devolved and also how unwilling the more senior layers of the org were to address it. At a whole company level, the rot really sets in when you start to lose the key people that built and know the system. That seems to be what’s happening here, and it does not bode well for MS in the medium term.
• acedTrex 7mo ago
  This is an insanely blunt look into some serious issues with microsoft.
• deleted 7mo ago
  [deleted]
• ludwigvan 7mo ago
  I had the misfortune of having to use Azure back in 2018 and was appalled at the lack of quality, slowness. I was in GitHub forums, helping other customers suffering from lack of basic functionality, incredible prices with abysmal performance. This article explains a lot honestly. Google’s Cloud feels like the best engineered one, though lack of proper human support is worrying there compared to AWS.
  • reddozen 7mo ago
    GCP's support is abysmal. Our assigned customer support agent has changed 3 times in as many months. it's really a dice roll if our quota increase requests are even acknowledged or we can get clarification on undocumented system limitations.
  • acdha 7mo ago
    I thought that about GCP until I used it more seriously and kept running into issues where they didn’t have some feature AWS had had for ages, and our Google engineers kept saying the answer was to run your own service in Kubernetes rather than use a platform service which did not give me confidence that they understood what the business proposition was.
  • hunterpayne 7mo ago
    Unless you work in Alphabet's marketing department, then no GCP isn't the best one. The most reliable cloud has always been AWS by a wide margin. The exec in charge of GCP has had to apologize in public on multiple occasions for GCP's reliability problems. Sounds like they have fixed them by now (years later) but that doesn't make up the disaster that was BigQuery. Also, GCP is more focused on smaller customers so perhaps that's the part that works for you. AWS can be a bit daunting. But AWS actually versions their APIs and publishes roadmaps and timelines for when APIs get added and retired and what you should use instead. GCP will just cancel things on short notice with no replacement.
  • pragmatic 7mo ago
    I still remember the day the disks dissappeared on the prod db running on gcp.
• Anon1096 7mo ago
  The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any. From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. There are certainly problems in Azure, but it's huge and rough edges are to be expected. It mostly marches on. IMO maturity is realizing this and working within the system to improve it rather than trying to lay out all the dirty laundry to an Internet audience that will undoubtedly lap it up and happily cry Microslop. Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? Azure is still chugging along apparently despite everything being mentioned. People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.
  • sabedevops 7mo ago
    He might sound like he has a grudge but you sound like you’re personally invested. Shill?
  • deleted 7mo ago
    [deleted]
  • kraemahz 7mo ago
    AWS and Google Cloud are both huge and are significantly better in UX/DX. My only experience with Azure was that it barely worked, provided very little in the way of information about why it didn't. I only have negative impressions of Azure whereas at least GC and AWS I can say my experiences are mixed.
  • jiggawatts 7mo ago
    > People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true. Or… you’ve just normalised the deviation. One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires. After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to wallow with everyone else that has become accustomed to the filth. As an Azure user I can tell you that it’s blindingly obvious even from the outside that the engineering quality is rock bottom. Throwing features over the fence as fast as possible to catch up to AWS was clearly the only priority for over a decade and has resulted in a giant ball of mud that now they can’t change because published APIs and offered products must continue to have support for years. Those rushed decisions have painted Azure into a corner. You may puff your chest out, and even take legitimate pride in building the second largest public cloud in the world, but please don’t fool yourself that the quality of this edifice is anything other than rickety and falling apart at the seams. Remind me: can I use IPv6 safely yet? Does it still break Postgres in other networks? Can azcopy actually move files yet, like every other bulk copy tool ever made by man? Can I upgrade a VM in-place to a new SKU without deleting and recreating it to work around your internal Hyper-V cluster API limitations? Premium SSDv2 disks for boot disks… when? Etc… You may list excuses for these quality gaps, but these kinds of things just weren’t an issue anywhere else I’ve worked as far back as twenty years ago! Heck, I built a natively “all IPv6” VMware ESXi cluster over a decade ago!
• lokar 7mo ago
  This reads pretty bad, and I believe it was. I worked on (and was at least partly responsible for) systems that do the same thing he described. It took constant force of will, fighting, escalation, etc to hold the line and maintain some basic level of stability and engineering practice. And I've worked other places that had problems similar to the core problems described, not quite as severe, and not at the same scale, but bad enough to doom them (IMO) to a death loop they won't recover from.
• yoyohello13 7mo ago
  I don't know if any of this is true, but as a user of Azure every day this would explain so much. The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get the services up who knows if they actually work as advertised. I'm honestly shocked anything manages to stay working at all.
  • mettamage 7mo ago
    Question: how would you compare it to AWS or GCP?
  • ragall 7mo ago
    We migrated some services to AKS because the upper management thought it was a good deal to get so many credits, and now pods are randomly crashing and database nodes have random spikes in disk latency. What ran reliably on GCP became quite unpredictable.
    • SeriousM 7mo ago
      Interesting! We're using AKS with huge success so far, but lately our Pods are unresponsive and we get 503 Gateway Timeouts that we really can't trace down. And don't get me started on Azure Blob Tables...
      • ragall 7mo ago
        In our case this was only a month ago, and now we're stuck because management thought it was a good idea to sign a hefty spend commitment.
        • a012 7mo ago
          In our case, we spent to much time of engineer time just to put up with Azure but there’s no good ROI. It took sometime for the upper management to realize Azure is shit and cut the cost
• pavlov 7mo ago
  The first couple of paragraphs felt like a parody of a guy who goes to a diner and gets upset the waitress doesn’t address him as Dr. It didn’t get any better.
  • lokar 7mo ago
    His writing style is fairly over the top (he is Swiss, and I have seen this before, but not most of the time), but most of the technical content seems true to me.
    • axelriet 7mo ago
      In all fairness, you are right :)
      • lokar 7mo ago
        IME, liechtensteiners are more Swiss than the Swiss
        • axelriet 7mo ago
          Nah ;)
• deleted 7mo ago
  [deleted]
• OldOneEye 7mo ago
  Some previous colleague of mine has to work with Azure on their day to day, and everything explained in this article makes a lot of sense when I get to hear about their massive rantings of the platform. 12 years ago I had to choose whether to specialize myself in AWS, GCP or Azure, and from my very brief foray with Azure I could see it was an absolute mess of broken, slow and click-ops methodology. This article confirms my suspicions at that time, and my colleague experience.
• abtinf 7mo ago
  So this is why GitHub is having so many problem…
• ok123456 7mo ago
  [flagged]
• _pdp_ 7mo ago
  The personal account makes a lot of sense, although I could easily see why the OP was not successful. Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator. The second thing is that this series of blog posts (whether true or not, but still believable) provides a good introduction to vibe coders. These are people who have not written a single line of code themselves and have not worked on any system at scale, yet believe that coding is somehow magically "solved" due to LLMs. Writing the actual code itself (fully or partially) maybe yes. But understanding the complexity of the system and working with organisational structures that support it is a completely different ball game.
  • okanat 7mo ago
    > Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator. I was thinking like this for a while but, now, I think this expectation is majorly false for a senior individual contributor. Especially when someone who can push out a detailed series of blogposts and has tried step-wise escalation. Communication is a two-way street. Unlike the individual contributors, the management is responsible for listening and responding to risk assesments by the senior members and also ensuring that the technical competence and experienced people are retained in a tech company. If a leader doesn't want to keep an open ear, they do not belong there. If there is a huge attrition of highly senior people from non-finalized projects, you do not belong leadership either. Both cases are mentioned in the article. Unfortunately our socioeconomic and political culture in the West has increasingly removed responsibilities and liabilities from the leadership of the companies. This causes people with lackluster technical, communication and risk assesment mentality being promoted into leadership positions. So outside of a couple completely privately owned companies or exceptionally well organized NGOs, it will be increasingly difficult to find good leaders.
  • grensley 7mo ago
    Absolutely textbook "Brilliant Jerk". Dude just whines and whines and whines. If you're so good, why can't you get anybody to work with you?
• arccy 7mo ago
  from part 2: > Worse, early prototypes already pulled in nearly a thousand third-party Rust crates, many of which were transitive dependencies and largely unvetted, posing potential supply-chain risks. Rust really going for the node ecosystem's crown in package number bloat
  • Aperocky 7mo ago
    It really is about time that somebody do something about it. Start with tokio. Please vend one dependency battery included, and vendor in/internalize everything, thanks.
    • okanat 7mo ago
      There is a difference between individual packages coming out of a single project (or even a single Cargo workspace) vs them coming out of completely different people. The former isn't a problem, it is actually desirable to have good granularity for projects. The latter is a huge liability and the actual supply chain risk. For example, Tokio project maintains another popular library called Prost for Protobufs. I don't think having those as two separate libraries with their own set of dependencies is a problem. As long as Tokio developers' expertise and testing culture go into Prost, it is not a big deal to have multiple packages. Similarly different components of the Tokio itself can be different crates, as long as they are built and tested together, them being separate dependencies is GOOD. Now to use Prost with a gRPC server, I need a different project: tonic which comes from a different vendor: Hyperium. This is an increased supply chain risk that we need to vet. They use Prost. They also use the "h2" crate. Now, I need to vet the code quality and the testing cultute of multiple different organizations. I have a firm belief that the actual People >>> code, tooling, companies and even licensing. If a project doesn't have (or retain) visionary and experienced developers who can instill good culture, it will ship shit code. So vetting organizations >> vetting indiviual libraries.
  • dralley 7mo ago
    Rust is nowhere close to Node in terms of package number bloat. Most Rust libraries are actually useful and nontrivial and the supply chain risk is not necessarily as high for the simple reason that many crates are split up into sub-crates. For example, instead of having one library like "hashlib" that handles all different kinds of hashing algorithms, the most "official" Rust libraries are broken up into one for sha1, one for sha2, one for sha3, one for md5, one for the generic interfaces shared by all of them, etc... but all maintained by the same organization: https://github.com/rustcrypto/ https://github.com/rustcrypto/ Most crypto libraries do the same. Ripgrep split off aho-corastick and memchr, the regex crate has a separate pcre library, etc. Maybe that bumps the numbers up if you need more than one algorithm, but predominantly it is still anti-bloat and has a purpose...
• deleted 7mo ago
  [deleted]
• vsgherzi 7mo ago
  I've said it before and I'll say it again. I'm glad rust has good package management I really am. However given that aspect, it ends up forming a dependency heavy culture. In situations like this it's hard to use dependencies due to the amount of transitive dependencies some items pull in. I really which this would change. Of course this is a social problem so I don't expect a good answer to come of this....
  • Aperocky 7mo ago
    Environment is part of the package management. As it stand, it's better than npm only because it is in rust.
    • jacquesm 7mo ago
      That bar is screwed to the floor.
• deleted 7mo ago
  [deleted]
• plantain 7mo ago
  I just do not understand how Azure has the scale it does. You only need to login and click around for a bit to see this is not a coherent system designed by competent people. Let alone try and actually build something on it. Who are the customers? Who is buying this shit?
  • whatever1 7mo ago
    The VPs who think that they got a good deal by combining with o365
  • zthrowaway 7mo ago
    Microsoft shops. Lots of C# devs gravitate to it naturally. I’m glad I abandoned the MS stack over a decade ago.
    • matt123456789 7mo ago
      .NET Core runs just as well on ECS though. And C# tooling is rock solid in VS Code on Mac. No need to touch Azure or Windows.
  • jmuguy 7mo ago
    From my old experience in IT - people just default to Microsoft for everything. They don't want to hassle with learning anything else and assume better the devil you know. Glad I'm out of that world but its wild what people will put up with.
  • okanat 7mo ago
    People and organizations that built things on top of Microsoft tech. Especially with a long history going back to NT times. HN, YC, startup environment or academia is a Unix bubble. They all feed into each other. Especially because Linux is gratis which helped all of those to deploy projects/products/papers cheaply. Unix systems traditionally lack much of the upper layers, so it is the responsibility of the company, persons, developers to deal with the OS minutea. You need sysadmins, devops, SREs. Those are common roles again in this Unix bubble. The dependency chains here are usually flatter since it keeps mid-term costs lower. Other organizations like governments and bigger orgs like banks prioritize having somebody else liable (i.e. they can blame) and they prefer to not hire technical competence in their orgs but rely on other companies. This is where Microsoft gets a lot of clients. You buy a bunch of server licenses. Your Microsoft support person installs them and installs IIS via GUI. And then you just upload your code every now and then. The OS updates, IIS server etc are all the responsibility of Microsoft and the middlemen companies. Minimal competence from the orginal org is required. There are multiple middlemen businesses who all give zero fucks about anything but whatever the immediate downstream from them. This is more usual in already publicly traded huge businesses. Moreover the investors actually mandate certain things that only this kind of layers of irresponsibility can deliver :) So you see this kind of switch happening towards IPOs. Azure is the cloud labeling and forcing the first paradigm over the second paradigm for Microsoft products. It got lots of support because shareholders liked it. I don't think the original NT design and Microsoft's business model was bad, it actually worked very well. However, shareholders gonna shareholder. So they pushed hard for Microsoft and their clients to move to the "cloud". Microsoft executives saw the huge profit and share value potential of pushing Azure the brand too. It was the AI of 2010s afterall.
• Frannky 7mo ago
  I was always very curious why people are using azure. Clunky difficult to setup and crazy prices. I know a person being very happy with them because of the credits they gave it to him. I felt I probably don't have a model that explains what is going on there and that would be cool to know why people pay them vs the competion
  • diamondage 7mo ago
    In my experience Azure endpoint versus openAI endpoint was way faster and significantly cheaper.
  • deleted 7mo ago
    [deleted]
• patrickRyu 7mo ago
  [flagged]
• gamblor956 7mo ago
  TLDR: It turns out that Nadella despite being an engineer is actually quite bad at managing engineering. Who would have thought?
  • tuan 7mo ago
    I thought he was a PM.
    • axelriet 7mo ago
      There is a video floating where he demoed Excel, and his title was Technical Marketing Manager. Maybe he pivoted to PM later? https://youtu.be/ddHDclRfzxM https://youtu.be/ddHDclRfzxM Base education, he’s an EE.
• shmoil 7mo ago
  [flagged]
  • Natfan 7mo ago
    pardon?
• outworlder 7mo ago
  Well, part 3 at least explains something I've observed; the platform is incredibly unstable. The same calls, with the same parameters, will often randomly fail with HTTP 400 errors, only to succeed later(hopefully without involving support). That made provisioning with terraform a nightmare. I won't even dive too much into all the braindead decisions. Mixing SKUs often isn't allowed if some components are 'premium' and others are not, and not everything is compatible with all instances. In AWS, if I have any EBS volume I can attach it to any instance, even if it is not optimal. There's no faffing about "premium SKUs". You won't lose internet connectivity because you attached a private load balancer to an instance. Etc... At my company, I've told folks that are trying to estimate projects on Azure to take whatever time they spent on AWS or GCP and multiply by 5, and that's the Azure estimate. A POC may take a similar amount of time as any other cloud, but not all of the Azure footguns will show themselves until you scale up.
• Manouchehri 7mo ago
  I've seen Azure OpenAI leak other customer's prompt responses to us under heavy load. https://x.com/DaveManouchehri/status/2037001748489949388 https://x.com/DaveManouchehri/status/2037001748489949388 Nobody seems to care.
  • mememememememo 7mo ago
    Did you anomomize those? Did Azure dox them or send the templated version?
    • Manouchehri 7mo ago
      Azure sent them to me like that. I only saw two companies mentioned in the messages I got back. I reached out to both to try to confirm, but never heard back.
  • jmogly 7mo ago
    This is insane, when you say azure OpenAI, do you mean like github copilot, microsoft copilot, hitting openai’s api, or some openai llm hosted on azure offering that you hit through azure? This is some real wild west crap!
    • nkozyra 7mo ago
      The latter, their arrangement with OpenAI enabled this.
    • pratyushnair01 7mo ago
      I have noticied a similar bug on Copilot. I noticed a chat session with questions that I had no recollection of asking. I wonder if it's related. I brushed it off as the question was generic.
      • Manouchehri 7mo ago
        I would guess that Copilot uses Azure OpenAI. In my small sample size of a bit over a 100 accidentally leaked messages, many/most of them are programming related questions. It's easy to brush it off as just LLM hallucinations. Azure OpenAI actually shows me how many input tokens were billed, and how many input tokens checked by the content filter. For these leaked responses, I was only billed for 8 input tokens, yet the content filter (correctly) checked >40,000 chars of input token (which was my actual prompt's size).
• deleted 7mo ago
  [deleted]
• throwawayslop12 7mo ago
  Power Platform is of the same quality, I’d avoid it if possible. I was a principal engineer in the Power Platform org and it always felt like a disorganized mess. Multiple reorganizations per year, changing priorities and service ownership.
  • Foobar8568 7mo ago
    These days, at work, I need to support applications build on Azure and Power Platform. Both are a hot mess. We get notifications that our APIM is down for at least 15min every weeks at random times. Power Platform is just a "preview" mess, things break and are not functional. I complained about it and basically was told to shut up, the industry is using them, so they must be right. No one is testing anything anymore.
    • jojobas 7mo ago
      It's a bit astounding to realize Ballmer was good.
    • truekonrads 7mo ago
      What were the issues behind "APIM down"?
  • snarfy 7mo ago
    Nothing has changed.
• thelastgallon 7mo ago
  > That entire 122-strong org was knee-deep in impossible ruminations involving porting Windows to Linux to support their existing VM management agents. > My day-one problem was therefore not to ramp up on new technology, but rather to convince an entire org, up to my skip-skip-level, that they were on a death march. > I later researched this further and found that no one at Microsoft, not a single soul, could articulate why up to 173 agents were needed to manage an Azure node This is most corporates. I'm sure this was celebrated as as a successful project and congratulations to everyone, along with big bonuses, RSU, raises, and promotions, mostly to other orgs to bring this kind of 'success' to other projects (or other companies). These people mostly are gone in less than 2 years. They continue to take 'wins'. The VPs are dumb as shit, but they need 'successful' projects that have fancy names that they can present to their exec team. The 173 agents are to give wins to a large number of people and teams, all these people contributed to this successful project. If it continues, there will be a lessons learned powerpoint, followed by 10x growth in headcount, promotions to everyone and double down. 270 people can deliver a baby in 1 day and all that.
  • thelastgallon 7mo ago
    In part 2 > This group was now tasked with moving their inherited stack to the new Azure Boost accelerator environment, an effort Microsoft had publicly implied was well underway at Ignite conferences since 2023. The goal is to attach your projects to something announced by the CEO and ride the career rocketship!
  • nijave 7mo ago
    Can almost guarantee there were 173 agents because there were 173 silo'd teams with competing goals and priorities working on their own codebases in isolation. And no, a 174th team doesn't solve it. Communication and collaboration across teams is key
• g_host 7mo ago
  "The company formalized the idea that defects could be fixed through human intervention on live production systems" Uh...yeah. I think we all realized that years ago.
• g_host 7mo ago
  "The company formalized the idea that defects could be fixed through human intervention on live production systems" (From Part 5). Uh...yeah. I think we all realized that years ago.
  • axelriet 7mo ago
    Great but then you tie your growth to the support people headcount. Normally you would see enormous costs upfront for R&D and bringing the thing up, then marginal costs when adding capacity (the hardware, mostly)—if capacity is proportional to the number of humans looking after the system, you will soon hit a limit, and the cost won’t look good either.
• kshri24 7mo ago
  The only time I used Azure was for setting up Microsoft as a provider for authentication. Put me through a never-ending loop of asking for a Government of India issued document that was already submitted. Human support was non-existent. Decided never to use Azure in any product after that horrible experience. If you cannot even get auth right I shudder to think what the rest of the product will be like to deal with should issues arise.
• bradleyankrom 7mo ago
  My most memorable anecdote from working in Azure is that they had two products named Purview and the internal MS people I talked to never figured out which one I was trying to use.
  • egorfine 7mo ago
    Astronauts have the same problem now.
• sakesun 7mo ago
  > Few engineers could reliably build the software locally I've just listened to Longhorn story on Monday and have heard the same thing.
  • _lvbh 7mo ago
    Could you link the story by any chance? I've been using Longhorn for a while and on one particular system, it has an odd tendency to corrupt XFS.
    • sakesun 7mo ago
      Probably this one https://www.youtube.com/watch?v=RpRZ8BQiiMo https://www.youtube.com/watch?v=RpRZ8BQiiMo
• purpleidea 7mo ago
  Microsoft Azure has always been a clown show. I've found so many obvious bugs. The quality is not there and never will be. No serious companies rely on it. Use virtually any other vendor or host it yourself.
• ryguz 7mo ago
  [dead]
• ChicagoDave 7mo ago
  I was a career Microsoft stack developer until Azure. Comparing it to AWS immediately forced me to make a decision to move away from their stack and towards AWS. Just the networking and security infrastructure was complete trash compared to how those things worked in AWS. Not one regret in my decision.
• jamwhite 7mo ago
  This read was a blast from the past. I'm not going to comment on much from OP and instead give a little of my experience there. Straight out of college in 2017 I joined the Compute Fabric Controller (FC) org as a SWE on an absolutely wonderful team that dealt with mostly container management, VM and Host fault handling & repair policies, and Fabric to Host communication with most of our code in the FC. I drove our team's efforts on the never ending "Unhealthy" node workstream, the final catch-all bucket in the Host fault handler mentioned in OP. I also did heavy work in optimizing repair policies, reactive diagnostics for improved repairs and offline analysis, OS and HW telemetry ingestion from the Host like SEL events into the repair manager in real time, wrote the core repair manager state machine in the new AZ level service that we decoupled from the Fabric, drove Kernel Soft Reboot (KSR)/Tardigrade as a repair action for minimal VM impact for some host repairs, and helped stand up into eventually owning a new warm path RCA attribution service to help drive the root underlying causes of reliability issues and feed some offline analysis back into the live repair manager. The work was difficult but also really really interesting. For example, Balancing repair policies around reliability is tricky. There's a constant fight in repair policies in grey situations between minimizing total VM downtime vs any VM interruptions/reboots/heals at all, because the repair controller doesn't have perfect information. If telemetry is pointing to VMs being degraded or down on the host, yet in reality they're not, we are the ones inducing the VM downtime by performing an impactful repair. If we wait a little while before taking an impactful repair action, it may be a transient issue that will resolve itself in the moment, at which point we can do much less impactful repairs after like Live Migration if the host is healthy enough. On the flipside, if some telemetry is saying the VMs are up yet they're down in reality and we just don't know it yet, taking time to collect diagnostics and then take a repair action(s) leads to only more overall total downtime. When I joined in 2017 our team was 7 or 8 people including myself, yet had enough work for at least double that amount of people. On-call was a nightmare the first 2 years. Building Azure back then was like trying to build a car while already sitting behind the steering wheel of that car as it was already barreling down the highway. Everyone on my immediate team the first couple years were a joy to work with, highly competent, hard working, and all of us working absurd hours. For me 60hrs/wk was avg, with many weeks ~80 and a few weeks ~100. Other than the hours though, it was a splendid team environment and I'd like to think we had good engineering culture within our team, though maybe I'm biased. Engineering culture and quality did however vary substantially between orgs and teams. We were heavily under resourced and always needed more headcount, as did nearly every other team in Azure Compute. That never changed during my tenure even though my team's size ballooned to ~20 by 2020, and eventually big enough to where we had to split the team. There was high turnover from the lack of headcount and overwork which was somewhat alleviated by lowering the hiring bar... which obviously opened up another can of worms. This resourcing issue might explain, in part, why Azure is the way it is. We were always playing catchup as a result of the woes of chronic understaffing for years. I eventually burnt out which turned into spiraling mental health, physical health issues, constant panic attacks, and then a full blown mental health crisis after which I took LOA and eventually left the company. I came back briefly for a bit during LOA, and learned that the RCA service I'd built with the help of a coworker (who also left Azure) and was only a small part of our overall workload, had turned into a full fledged team of 9 people dedicated to working on that service in my absence. I know that stating some of this might affect my employment in the future but I don't really care. I know I'm not alone in experiencing burnout working in Azure. It wasn't my manager's fault either, he was amazing. He'd often ask and I would incorrectly yet confidently reassure him that I wasn't burning out but I simply didn't notice the signs. Things are better now though and I'm just happy to be here. Kudos to the many brilliant people I worked alongside there, I hope you're all doing great.
• xyst 7mo ago
  til: there’s individuals/people that "trusted" azure at all I only used that shit platform because some Microsoft consultant convinced idiotic C-suite that Azure was the future.
• jimbobimbo 7mo ago
  When things must be shipped quickly, shit breaks and corners are cut; large orgs are full of disfunction. Not sure if such insight was worth of setting your own career on fire.
• Yoofie 7mo ago
  > Few engineers could reliably build the software locally; debugger usage was rare (I ended up writing the team's first how-to guide in 2024); and automated test coverage sat below 40%. A key clue and explains why so much of what Microsoft puts out is garbage. Wow.
  • markus_zhang 7mo ago
    TBH user as beta tester is probably an official policy. Why debug when someone debugs for you?
• deleted 7mo ago
  [deleted]
• CalRobert 7mo ago
  What makes anyone start a new project and think “I know, I’ll use Azure!”? I really don’t get it. Do they have a great sales org? Is it because a phb thinks “well they made Office so it must be good”? I interviewed with a Dutch energy company migrating infra from AWS -to- Azure and I have no idea what would make them do that (aside from inertia, but then why use Azure in the first place?) And for some reason Azure usage is rampant in Europe.
  • irusensei 7mo ago
    Companies coming from Active Directory and Office.
  • jstanley 7mo ago
    They give free credit to startups if you fill in a few forms.
    • CalRobert 7mo ago
      so does AWS and GCP... but pretty bad if that's the deciding criteria.
  • exac 7mo ago
    At one startup I was in, Azure sales proactively reached out to the CEO on LinkedIn and then we were urged to swap off to it.
  • derwiki 7mo ago
    At the startup I worked at in 2023, Azure was considered the only “safe” way to use OpenAI APIs in prod (eg agreements that the data couldn’t be used for training). Working with Azure was one of the worst parts of that job.
  • slyall 7mo ago
    Where I live (New Zealand) Microsoft is a much larger percentage of IT infrastructure than say Bay Areas startups. Companies are already used to working with Microsoft. Building on Microsoft's cloud feels natural.
• branko_d 7mo ago
  I think this is especially problematic (from Part 4 at https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion-2f5 https://isolveproblems.substack.com/p/how-microsoft-vaporize...): "The team had reached a point where it was too risky to make any code refactoring or engineering improvements. I submitted several bug fixes and refactoring, notably using smart pointers, but they were rejected for fear of breaking something." Once you reach this stage, the only escape is to first cover everything with tests and then meticulously fix bugs, without shipping any new features. This can take a long time, and cannot happen without the full support from the management who do not fully understand the problem nor are incentivized to understand it.
  • idorosen 7mo ago
    Or to simplify the product and rebuild.
    • axelriet 7mo ago
      Exactly. But he’s right about management, first the problem must be acknowledged and that may make some people look bad.
    • teeray 7mo ago
      “Rebuild” is also a four-letter word though at this stage too. The customer has a panel of knob-and-tube wiring and aluminum paper-wrapped wire in the house. They want a new hot tub. They don’t want some electrician telling them they need to completely rewire their house first at huge expense, such that they cannot afford the hot tub anymore. They’ll just throw the electrician out and get some kid in a pickup truck (“You’re Absolutely Right Handyman LLC”) to run a lamp cord to their new hot tub. Once the house burns to the ground, the new owners will wire their new construction correctly.
  • gherkinnn 7mo ago
    Once you reach this stage, the only escape is to jump ship. Either mentally or, ideally, truly. You're in an unwinnable position. Don't take the brunt for management's mistakes. Don't try to fix what you have no agency over.
• nosefrog 7mo ago
  We run 1000s of machines in Azure. It's garbage. Very few features work. Nodes are always having strange issues, especially on the networking side. And the worst part is that Azure support has 0 interest in actually debugging things. We just got out of an outage today caused by the insanely slow SSDs that they attach to their postgres dbs by default.
• Dansvidania 7mo ago
  I have been in a Microsoft adjacent company (meaning lots of people bounced to and from Microsoft to it) and all this makes a lot of sense. The almost ideological “everything in house” and politically oriented philosophy they had fits like a glove. Some of the ex Microsoft people hated it, some of them missed it. But the picture they made was pretty bleak. Given how windows is going what’s described in the article doesn’t seem so shocking either. Even though they need not be correlated products, I can’t help but seeing a similar shortsightedness in the playbooks they are adopting.
• JackSmith_YC 7mo ago
  [dead]
• elankart 7mo ago
  I've worked in Windows for many many years, no idea who this guy is. He is randomly name dropping. He wants attention.
  • pas 7mo ago
    and what's with the parents? for running containers?
• petterroea 7mo ago
  A business man at a prior employer sympathetic with my younger, naive "Microsoft sucks" attitude told me something I remember to this day: Microsoft is not a software company, they have never been experts at software. They are experts at contracts. They lead because their business machine exceeds at understanding how to tick the boxes necessary to win contract bids. The people who make purchasing decisions at companies aren't technical and possibly don't even know a world outside Microsoft, Office, and Windows, after all. This is how the sausage is made in the business world, and it changed how I perceived the tech industry. Good software (sadly) doesn't matter. Sales does. This is why most of Norway currently runs on Azure, even though it is garbage, and even though every engineer I know who uses it says it is garbage. Because the people in the know don't get to make the decision.
  • szundi 7mo ago
    [dead]
  • dbdr 7mo ago
    But that also means that if you as a user/customer can make choices based on technical merits, you'll have a significant advantage.
    • lmm 7mo ago
      An advantage how? Maybe you'll have one or two more 9s of uptime than your competitors; does that actually move the needle on your business?
      • Aissen 7mo ago
        Why wouldn't it move the needle? Less time spent, less frustration, more performance, more resources focused on the business?
      • hunterpayne 7mo ago
        The biggest expense in software is maintenance. Better software means cheaper maintenance. If you actually want to have a significant cost advantage, software is the way to go. Sadly most business is about sales and marketing and has little to do with the cost or quality of items being sold.
• momo_dev 7mo ago
  i run fastapi APIs on linode with cloudflare in front and honestly the simplicity is underrated. predictable billing, docs that match reality, no surprise platform regressions. for a straightforward API workload the hyperscaler tax doesn't make sense unless you genuinely need their scale
  • Terretta 7mo ago
    Hate to break it to ya, you picked an emerging hyperscaler: https://www.sdxcentral.com/news/cloudflare-has-the-edge-in-hyperscaler-fight-with-booming-q4-results/ https://www.sdxcentral.com/news/cloudflare-has-the-edge-in-h...
    • nijave 7mo ago
      Linode is also owned by Akamai...
    • momo_dev 7mo ago
      i guess the difference is i chose my hyperscalers à la carte instead of getting the all-in-one bundle. at least when cloudflare breaks something i can still ssh into my linode and debug it directly
• h6d_100c 7mo ago
  This makes it extra silly to trust that Github won't train on your private repos, if they haven't already - just by accident
• physhster 7mo ago
  This reads like Google culture too...
• physhster 7mo ago
  Thanks for that, now I have a rock-solid argument when people say "oh we're already Microsoft customers, we'll just use Azure, it's easier, and they have Active Directory!!"
• xxxboxxx 7mo ago
  We signed up to go all-in on Azure because our CEO got an xbox to take home to his kids.
• alex0ptr 7mo ago
  I’ve been working with Azure and Azure Germanyfor the past years and have a strong history with AWS. I cannot count how many times disks were not attaching during AKS rescheduling. We build polling where we polled Entra Id for minutes until it became “eventually” consistent - not trusting a service principal until it was fetched at least one minute consistently. The slowness of Azure Functions was unbearable. On Azure germany IoT Hubs had to be “rebooted” by support constantly - which was a shocking statement in itself. The docs always lying or leaving out critical parts. The whole Premium vs Standard stuff is like selling windows licenses. The role model and UI is absolutely inconsistent. The stability, consistency of IAM, and speed of AWS in comparison makes me truly wonder how anyone stays with Azure. One reason might be that the Windows instances are significantly cheaper though..
• debarshri 7mo ago
  Substack is having its moment. First, deepdelver, now this.
• einrealist 7mo ago
  Axel's engagement with the issue and refusal to give up is admirable. It also demonstrates that code and architecture remain important even in an era when managers believe these subjects can now be handled by LLMs. Imagine if LLMs were mandated for use in such an environment, further distancing SWEs from the code and overarching architectural choices. I am not saying that it can't work. But friction and maturity through experience really matters. Also explains perfectly why I never met an engineer who was eager to run workloads on Azure. In orgs I worked, either the use of Azure was mandated by management (probably good $$ incentives) or through Microsoft leaning into the "Multi-Cloud for resilience" selling point, to get Orgs shift workloads from competitors. Its also huge case for open (cloud) stack(s).
• egorfine 7mo ago
  > Cutler’s intent was to produce a system with the same level of quality, unshakable reliability, and attention to detail he was famous for in his work on VMS and NT. I'm not sure whether this is serious or irony.
  • axelriet 7mo ago
    Search VMS stability, I think the consensus is clear. Then Google VMS longest uptime, and the record is 28 years. VMS often achieved five nines over 10 years (99.999%) so no irony. He took a bunch of folks with him from DEC to Microsoft to make NT, and of course his principles. Nowadays NT is bomb-proof believe it or not. Most of the crashes are in device drivers and some rare times in the UI code (Win32k) that should not be there, but the kernel itself is solid. (Yes I am a big fan)
    • egorfine 7mo ago
      VMS, yes. No doubts. NT, no. Again, no doubts.
    • deleted 7mo ago
      [deleted]
    • markus_zhang 7mo ago
      I remember reading "Showstoppers" and David was quoted to say "If you break the build I'm the lawn mower and your ass is grass". Do you think such attitude is mandatory for good kernel level code? (I actually think it does and argued with people on HN, although I never wrote any professional kernel code myself)
      • EvanAnderson 7mo ago
        Aside: If you liked "Showstopper" you should give a listen to the Computer History Museum's oral history interview with David Cutler: https://www.computerhistory.org/collections/catalog/102717163 https://www.computerhistory.org/collections/catalog/10271716...
• summitwebaudit 7mo ago
  [dead]
• egorfine 7mo ago
  > was maintaining in-memory caches containing unencrypted tenant data, all mixed in the same memory areas, in violation of all hostile multi-tenancy security guidelines Splitting caches to different isolated memory areas will not make shareholders happy, will not lead to promotion and will not even move the project forward. Simply put, designing secure software is detrimental in that environment.
• auggierose 7mo ago
  I tried to use Azure once (more than 5 years ago), and the signing up kept crashing on me for hours. Never used it again since then. Some things are obvious.
• luke5441 7mo ago
  On a leadership level it seems problematic that they ghosted the feedback. Direcly this leads to people like Axel who feel ownership of the problem to break NDAs and create company harming posts. In my experience they at least respond with corp speak platitudes meaning that they got the feedback and don't understand it or ignore it, but have been taught to always ask for feedback and answer it (but incentives are to ask for feedback, then ignore it).
  • axelriet 7mo ago
    To be honest, I don’t think this is “company harming”—what would be harming is Azure being pwned if they didn’t know and did nothing, or failing SLA at the wrong time. Now they know.
    • mike_hearn 7mo ago
      Azure has been repeatedly hacked very severely, and it doesn't seem to make much difference to their adoption.
    • luke5441 7mo ago
      The ultimate goal is to make customers spend money on Azure. Of course the information you published may make customers less likely to choose Azure, harming Microsoft. Being pwned can be explained away as an attacker having spent a lot of ressources to do so. Failing SLAs can be a calculated gamble. I myself am grateful you published this! It gives a great inside view on what is going on in big tech in general and Microsoft specifically.
      • axelriet 7mo ago
        Well, what you describe is plausible, but it would not be a good long-term strategy and is certain to backfire badly at some point. Then imagine your systems are key support systems with deep implications in government and the military, and the path you outline is not acceptable. Onboarding new customers on a sinking ship is dishonest at best, criminal at worst. So yes, I maintain that it helps more than it harms.
• goodpoint 7mo ago
  "isolveproblems", really?
  • axelriet 6mo ago
    "goodpoint"
• manmal 7mo ago
  For some reason, MS is still doing well. I’m not sure what conclusions I should draw from that, other than big businesses are hard to kill?
• nalekberov 7mo ago
  At this point, it’s very clear that people nowadays choose Rust mostly to be part of the cult rather than clearly understanding its shortcomings and advantages over languages such as C, C++. It has gotten to the point that some devs after watching a YouTube video criticizing C++ for two hours, announce C++ the worst programming language. Unfortunately, such people become decision makers at giant tech companies too.
  • deleted 7mo ago
    [deleted]
• andyjohnson0 7mo ago
  This reads like it was written by the Cleverest Person in the Room. I have to use Azure Devops at work, and some of the critique of Azure rings true for me, but the author-centric presentation was quite off-putting.
  • axelriet 7mo ago
    Sorry you felt like that.
• deleted 7mo ago
  [deleted]
• breppp 7mo ago
  I highly sympathize with the author and as a former user of Azure I agree it's a terrible mess. However, the author has committed magnificent career suicide. If you are in a dysfunctional environment you don't go from issue to issue and escalate each one, proactively finding problematic issues. You rather find the underlying issues (e.g. crashes not assigned) prioritize them and fix them. By constantly whistle blowing on separate issues to as high as the board, he is not trying to improve by evolution but by revolution and in revolutions heads roll
  • axelriet 7mo ago
    The timeline and facts were quite different. Debating an org-wide quality issue on a 100+ member team's alias is not whistleblowing.
• TomMasz 7mo ago
  The "no one at Microsoft, not a single soul, could articulate why up to 173 agents were needed to manage an Azure node" really stuck with me. You have to wonder how many other parts of the code lack ownership and are in there just because no one knows what will happen if you take them out.
  • mananaysiempre 7mo ago
    This reminds me of discussions of the “MinWin” initiative back in the Windows 7 timeframe, and how the obstacle was that nobody actually knew what you could take out of Windows and still have it work, so they had to be conservative.
• mathgeek 7mo ago
  Reading through this reminded me of just how engulfed in acronyms and lingo MS engineers must be. Much like AWS engineers with an acronym for every service that gets thrown around with the assumption of understanding, I felt like I needed a dictionary of those just to understand what was going on.
• mrsmrtss 7mo ago
  I see that it's fashionable to bash everything MS related in HN, but let's not pretend that the other major cloud providers don't have their own problems (e.g. https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f77... or https://blog.barrack.ai/google-gemini-api-key-vulnerability/ https://blog.barrack.ai/google-gemini-api-key-vulnerability/). We have had a couple of critical services hosted on Azure over ten years already, call me lucky, but we haven't had any major incidents. That said, the AI Foundry side is broken garbage at the moment, but so is also AI stuff from other providers.
  • nijave 7mo ago
    Their VMs and load balancers mostly work. Their managed services are a crapshoot. We routinely "self hosted" at the company that used Azure to ensure some semblance of stability. For instance, our Patroni clusters were much more performant and stable than Azure Single Server Postgres (what a terrible product...).
    • mrsmrtss 7mo ago
      Maybe this is why they retired Single Server PostgreSQL and are now offering only the new Azure Database for PostgreSQL (flexible server). Zero problem with the latter for us so far.
• mif 7mo ago
  I’m not an expert and surprised by the extent of Azure’s technical debt and its consequences. What would be a “minimal” reproducible configuration or setup of services that shows those technical deficiencies in the clearest way? A “benchmark for cloud computing services”, for a lack of a better description.
• torginus 7mo ago
  Personally, when asking others about their opinions on various cloud providers, AWS tends to emerge head and shoulders above the rest for one simple reason - AWS works. And the reason AWS works is that AWS runs on AWS (in stark contrast to Azure and GCP which afaik is not what MS and Google use internally). And when AWS doesn't work, support is there to help you. To add nuance to this statement, the other providers have their own strengths and standout features, but if you have to approach every single one of their features with suspicion that means you wont build stuff on top of them.
  • nijave 7mo ago
    I've also noticed AWS tends to have less "magic" global services and tends to favor cell architecture with partitions and isolation. These super duper magic global services seem to be the cause of most outages since the blast radius is so huge. On the other hand, the proposition of a magic, infinitely scaleable service endpoint is nice from a developer perspective.
    • torginus 7mo ago
      Even on AWS, if you go for the managed magic version of the thing, they'll make you pay more, lose some flexibilitym and the relinquished control will change things in a way that benefits AWS (slower scaling, limitations, unnecessary overprovisioning, overhead). An example - if you scale things manually by provisioning and starting EC2 instances via API - it will be more performant and cheaper than either Lambda or ECS Fargate (or Batch...). But those things at least work reliably. With the other two cloud providers, you'll likely run into a bug you cannot fix yourself, and you will have no support to help you.
• pjmlp 7mo ago
  Having now read the six parts, I assume the same management issues, and junior devs all over the place, are the reason why Windows development has become a mess, and Project Reunion went nowhere sane, leaving only Windows employees to care about WinUI 3 and WinAppSDK. If only we had a return of netbooks, meaning OEMs finally embracing GNU/Linux on consumer stores, instead of being left to technically minded aware of online stores.
• deleted 7mo ago
  [deleted]
• stego-tech 7mo ago
  I knew Microsoft was incredibly dysfunctional (you have to understand this if you're supporting their suite and want to succeed), but damn, I'm floored by the incompetence reported on from juniors to the Board and seemingly every step of leadership in between. Yet I'm also not surprised, because I keep encountering it in non-Microsoft orgs. The current crop of leadership in general seems to be so myopically focused on GTM and share price bumps that even the mere suggestion of a problem is a career-ending move for whoever reported it (ask me how I know). Making matters worse is that Boards and shareholders have let them get away with this for so long, across every major org, that these folks believe in their heart and soul that they're absolutely, infallibly correct. The higher up someone is in an organization, the higher the likelihood they'll reject any and all feedback from "beneath" them that is contrary to their already-decided-upon agenda. The kicker is that I'm not sure how to actually deal with this in a way that minimizes pain. In my subjective experience, these sorts of companies simply do not change until and unless there's literally no other option other than failure - and then, they're likely to choose failure for the parachute selection instead of doing the hard work of reform. Maybe what's needed is for Microsoft (or any of the legion of similarly dysfunctional enterprises out there) to genuinely fail in a non-recoverable way so as to shock the wider industry/economy into taking serious action on corporate misgovernance. Maybe failure is the best option. I don't know. I just know that this isn't tenable.
  • pjc50 7mo ago
    > Maybe what's needed is for Microsoft (or any of the legion of similarly dysfunctional enterprises out there) to genuinely fail in a non-recoverable way so as to shock the wider industry/economy into taking serious action on corporate misgovernance. The naive model of capitalism says that the benefit of market competition is that it's possible for failing companies to get out-competed by non-failing ones. In practice, there's enough of a combination of "natural monopoly", lock-in effects, and anti-competitive practices that the software landscape is covered in companies that are too big to avoid, let alone too big to fail.
    • stego-tech 7mo ago
      That's what I've been trying to impart on folks for a decade, now. The lack of regulations has let apex predators capture the environment, and short of an environmental collapse (as in, the sudden and permanent destruction of compute in general that makes their business unrecoverable), the only solution is hunting the hunters - i.e., government regulations, monopoly breakups, market penalties, etc. There is no feasible way for someone to out-compete Microsoft, Apple, Google, or Oracle. None. They have to fail in some capacity to a significant, global-economy-harming degree to even provide an opening to competition in the marketplace. Even if AI turned out to be a huge nothingburger tomorrow, they'd still be unassailable. That is the problem.
• dec0dedab0de 7mo ago
  Using Azure has severely affected my mental health over the last year. Reading these comments has been therapeutic.
• dzonga 7mo ago
  you have a current US president who has never read the Art of War. likely most company leadership, besides Hedge fund managers, have never read the Art of War either. this results in management that lacks a strategic focus - they want to win the next battle (down in a valley, while giving the enemy an upper hand to be on the hill). your infantry (low-level ICs) are smart and capable - and the org is actively pursing means to deskill them via some shit called (A.I) - your colonels (mid-management) are comfortable in their laurels since anyone who raises a voice is shown the door (hell most of them manage people now & don't fight anymore) then you wonder why the country, the org is losing. but hey at least we posted a massive valuation.
  • steve1977 7mo ago
    Why read the Art of War when you wrote (or had ghost-written for you) the Art of the Deal?
• sidewndr46 7mo ago
  Some of this reads like parody, for example: "Cutler’s intent was to produce a system with the same level of quality, unshakable reliability, and attention to detail he was famous for in his work on VMS and NT." I'm not really here to take shots at Dave Cutler, but Windows NT was not known for it's unshakeable reliability. If it's known for anything, it is known for lacking any basic security measures. I remember demonstrating to people who joined my WiFi network that I could automatically obtain remote shells on their laptop.
  • deleted 7mo ago
    [deleted]
  • EvanAnderson 7mo ago
    > I'm not really here to take shots at Dave Cutler, but Windows NT was not known for it's unshakeable reliability. NT itself (the kernel and native mode APIs) is pretty well designed and implemented, in my opinion. I know there were findings from fuzzing kernel and native mode APIs in early versions of NT, but by about the Windows 2000-era it was pretty solid. Win32 and the grown-up mess of APIs around it I'm less enthused with. NT itself is very impressive to me. My fever-dream OS is an NT kernel with a modern and updated Interix subsystem as the main subsystem, with Win32 as a compatibility layer.
• kangraemin 7mo ago
  "Risk aversion preventing fixes" is the most accurate part. I've seen this at other large companies too. You have a known bug, you know exactly how to fix it, but nobody will approve the change because "what if it breaks something else." So the bug stays forever and everyone just works around it. The irony is that the workarounds eventually cause more breakage than the fix ever would have.
• dgellow 7mo ago
  All those discussions about career suicide. Are you all that afraid to do what you think is right because you could get fired? What Axel does by coming public with his named attached is remarkable. He gains a lot of respect in my book. Even if it is one sided and details are missing
  • CobrastanJorji 7mo ago
    The career suicide wasn't escalating (although probably that was job suicide). The career suicide is venting and airing all of your former employee's dirty laundry. Unless your former employer is doing something deeply unethical, writing hit pieces against them after you leave is going to make you less attractive to future employers. Before this article, employers would see "experienced and available cloud engineer." After this, employers would see "backstabber who was probably fired for being a pain." But also, this is Hacker News. Many of us work for companies that are largely making the world worse in exchange for large salaries. Many of us have, probably unconsciously, built our lives around not doing what we think is right in exchange for not getting fired.
    • axelriet 7mo ago
      You have all the right words but some are in the wrong order.
    • axel479343 7mo ago
      that's a boomer ass take the world doesn't work like this anymore
• unsubtlecoder 7mo ago
  Explains GitHubs terrible uptime I guess …
• CodeCompost 7mo ago
  Azure Functions have been solid for us. No real weird downtimes and if something happens its usually because we did something wrong. We don't do very complicated things, mainly App Services with Azure SQL and Azure Functions. Having said that, Microsoft did botch the .NET 8 -> .NET 10 migration for Azure Functions with Consumption Plan. So yeah ... we're beginning to see some of the cracks.
• ninjagoo 7mo ago
  A tale as old as corporations. Corporate Ladders optimize for Ladder-Climbers, rather than Management Skills or Technical Skills. Organization Design is tough. And gets even more challenging with size. Unfortunately, Org Design over time falls to those folks that rose up the ladder, rather than folks dedicated to understanding and designing orgs. Switching from a Traditional org to an Agile one doesn't eliminate the need for thoughtful org design, it just changes the structures and incentives, and understanding and leveraging the interplay of various factors still requires unbiased organizational skills. Mature companies will often send executives through training around organization design, but separating out the incentives that apply personally to the executives, from what they do for the company, can be challenging. So larger companies will tend to have a org design or operating model team, and very large companies will formalize this as CoE or Transformation Offices. Still, getting that balance right can be tricky. Looks like MS failed badly in this instance. Maybe they learned from it, maybe they didn't. Judging by the way things are going with Win11, and the lack of response from the EVP, CEO and Board levels, maybe they ignored their internal folks that help with alignment, or more likely, simply laid them off!
• botulidze 7mo ago
  So GitHub migrating entirely to Azure is going to reduce the number of outages they are experiencing, right? Right??
  • joezydeco 7mo ago
    Nope. https://bsky.app/profile/justingarrison.com/post/3mig2gikibs23 https://bsky.app/profile/justingarrison.com/post/3mig2gikibs...
• robshippr 7mo ago
  The "too risky to deploy" problem is really a visibility problem. When you can't quickly see what's actually changing in a deploy, fear becomes the default. The teams that break out of this aren't the ones who stop shipping, they're the ones who build better signals before the deploy so engineers can ship with confidence instead of just hoping nothing breaks.
• Hasz 7mo ago
  As far as I know, you still cannot rename a resource. Insanity. I don't even work with it that much and have a laundry list of complaints about the weird little edge cases or funky pieces of documentation required to make things work.
• TheNewsIsHere 7mo ago
  This write-up is a shining example of why I’ve been rebuilding my business slowly away from Microsoft technology. Entra as IdP is one of the last projects. I’m probably not going to escape Exchange Online, but I’m going to be happy to finally federate the tenant to our internally managed IdP. My spouse’s employer mandated that everyone move off AWS “because they’re a competitor” (they’re absolutely not), and Microsoft was happy to roll out discounts for Azure. To say that has gone poorly would be generous. Azure is impressive in its own right, but it’s not comparable to AWS. (Which has its own problems, to be clear.) The stagnation in Azure is apparent everywhere you look. The capacity issues have only gotten worse. There are still change advisory callouts in the Azure Portal with dates in the year 2020.
• markus_zhang 7mo ago
  This is pretty damning, if half of it is true. I don’t work at Microsoft and I don’t have the knowledge to judge the reliability of Azure, but I do have friends who work as users of Azure and the words are not kind, especially the new Fabric database which is said to be crazy to pick for production at this stage — while MSFT switched the certification to Fabric already, pushing its customers to use it. I’ll never work in a company that uses Azure as its main cloud services, just for the sake of quiet nights. I do wonder what does it look like inside AWS and GCP, though. Is it the same level of chaos, but just because they started early they got more success? If that’s the case, maybe we can conclude, that very large cloud operation is not sustainable under the current company structure — because either the technical knowledge required is too dense, but companies won’t be able to retain workers, or because companies are forced to join the horn of the marketeers, eventually.
  • panzagl 7mo ago
    This article is like a cockroach in a restaurant dining room. Azure has one, GCP/AWS does not.
    • markus_zhang 7mo ago
      Awaiting for the AWS/GCP one...
    • axelriet 7mo ago
      The thing with cockroaches is that if even a single one is seen in the dining room and someone calls environmental health, regardless of the restaurant's prestige, they close it with immediate effect until they get their act together and a food sanitation inspection clears them. At the end, everyone feels better, in particular the customers.
• pwarner 7mo ago
  My comment history here is full of complaints about MS Teams, the chat app. It suffers from the "re-use every existing MS tech" problem. Building it on top of SharePoint I'm pretty sure resulted in it's top problems over the years (some fixed): - search sucked - can't scroll back to old messages - couldn't do private channels - the very concept of 1:1 teams to SharePoint site, resulting in a millions teams when all you really wanted was a channel - can't rename teams or channels - couldn't do private channels I'm sure many more I didn't catch. These are all observations from outside, I've never worked at MS
• barelysapient 7mo ago
  Is it just me or does this describe most of Microsoft software at the moment? I tried to sign into my personal microsoft account to setup an oauth flow and I was greated by an infinity repeating error dialog about some internal service that had failed. At work, I use outlook. The number of times I've gotten caught in an auth loop where I enter again and again my creds + tfa only for the screen to flicker and start all over again. Complete garbage.
• gigel82 7mo ago
  I have no doubt Azure sucks, but almost all huge projects like that have systemic issues. Axel sounds like a pretty smart guy, but wanted to point out I've seen this kind of behavior before, often from mid-level "job-hopping" engineers (sometimes with overly inflated egos) that overconfidently declare everything the organization is doing is BS and they have the magic solution to it. And yes, sometimes by sending long winded emails to very large internal groups about how their solution will address all the problems if only someone recognize their genius (and eventually give them a VP title and budget). Some of the time, they are well intended but missing crucial historical knowledge about why things are in the state they are and why what they're proposing was tried 5 times before and failed.
• solguarddev 7mo ago
  [flagged]
• tonylewislondon 7mo ago
  [dead]
• palmotea 7mo ago
  > Furthermore, I contributed to brainstorming the early Overlake cards in 2020-2021, drafting a proposal for a Host OS <-> Accelerator Card communication protocol and network stack, when all we had was a debugger’s serial connection. I also served as a Core OS specialist, helping Azure Core engineers diagnose deep OS issues. What exactly are these "Overlake accelerator cards"? What are they accelerating?
  • layer8 7mo ago
    Courtesy of Google: https://glennklockwood.com/garden/Azure-SmartNIC https://glennklockwood.com/garden/Azure-SmartNIC
• phillipcarter 7mo ago
  This smells of someone's Clawd writing something deceptively, much like the other semi-viral content that landed on reddit related to DoorDash systems.
• jrmg 7mo ago
  If, like me, you started reading and after a while started thinking “Wait, how many parts is this going to be in‽”, the answer is “six”.
• LeoStehlik 7mo ago
  Back in 2011 at Fujitsu, I ran one of the earliest Azure production subscriptions outside Microsoft. Windows Azure, mid-2011. I've watched this platform for 15 years from the outside. Part 1 barely scratches the surface. Read parts 2 through 6. The 173 agents story, the 200 manual node interventions per day, the WireServer sitting on the secure host side with unencrypted tenant memory mixed in shared address space, the letters to the EVP, the CEO, the Board - not a single acknowledgment. The most damning thing in this series ... except for technical debt ... is the silence at the top when someone handed them the diagnosis on a plate. Cutler's original vision was "no human touch." The gap between that and what Azure actually became is where the trillion dollars went. Go read the rest. It's worth it. Meanwhile on LinkedIn, there are still comments how adorable Microsoft leadership under Satya is... a carefully crafted PR image.
• drob518 7mo ago
  Great series of articles and completely believable. My first thought after reading is I hope the author doesn’t get sued for violating his non-disclosure agreement.
• gitowiec 7mo ago
  We say in Poland that the fish rots from the head down!
• throwaway0703 7mo ago
  Throwaway since I may want to work at Microsoft again one day. Given my own experience at Azure I believe all of this. The post demonstrates there are serious management and structural issues throughout a large part, if not all, of the organization. And it definitely sheds some light on my experiences with the networking platform being so fragile and unreliable. This post lends credence to the idea that large companies only care about security just enough to either not get compromised, or “just” to get mildly compromised. Defense in depth costs too much in management’s eyes, and they consider it a more wise use of resources to patch the holes after they’re made rather than prevent them in the first place. Thanks to the author for sharing, and I hope your subsequent role is more enjoyable. It feels like the only way to make the structural changes being suggested is to climb the corporate ladder to accumulate sufficient power plus social and political capital, and then get buy in to painstakingly steer that behemoth of an organization in a safer and more sustainable direction.
• philipallstar 7mo ago
  When the CEO is a product person and you need a platform:
• markus_zhang 7mo ago
  I have always wanted to find some technical refutes, and I found one on reddit. https://www.reddit.com/r/programming/comments/1sbir8j/comment/oe5axxq/?context=3 https://www.reddit.com/r/programming/comments/1sbir8j/commen... I'll skip the other comments and focus on the technical ones: > there are hundreds of “agents” which run on a one time basis to install systems as part of deployment architecture. These agents often amount to pretty simple scripts or programs. They most often run one time per update deployment, or if nodes are repaved. Some install small daemons. It’s called micro service architecture. Guy claims to be some cloud wiz but doesn’t get these basics. > That said He’s put cutlers original work on a pedestal, when fabric controller should have been replaced a decade ago. The monolithic nature of fabric has been a huge issue for reliability and scalability, and the company is trying its hardest to move as many features out of it into microservices as it can. I'm wondering if OP can answer this refute? Looks like the person is working in a neighbor team. No offense intended but I'm really curious about the technical part.
• belviewreview 7mo ago
  The problem started because Azure was initially designed and released in a huge rush because Microsoft was so far behind AWS and needed something better. I am reminded of the research finding that every human-designed complex system that works well started with a simple system that did did just one thing well, and new functions were added one at a time, with each one perfected before moving on to another. Which is the exact opposite of what happened here.
• deleted 7mo ago
  [deleted]
• phkamp 7mo ago
  To me that text reads a lot like an affidavit supporting a Qui Tam suit ?
• razkaplan 7mo ago
  [dead]
• wenbin 7mo ago
  Does this mean AWS's #1 position is safe? Every big cloud provider has its share of UX/stability/customer support issues. At this point, it feels less like AWS is the 'least bad' option because alternatives are even worse.
• bg24 7mo ago
  I read all 6 posts. OP is genuine and very talented. Unfortunate how foundational issues fall through the cracks in orgs.
• dryarzeg 7mo ago
  Now you can share this with anyone who says, “AI will make software faulty/buggy/unstable/garbage/whatever” - it won’t; people are perfectly capable of handling it themselves. /s
• mleonhard 6mo ago
  [dead]
• montjoy 6mo ago
  Is there a point where you step back and let it blow up so that management HAS to take the issues seriously? In other words, if you care, you might get enough minor fixes through to keep the product limping along instead of letting it become a big enough bonfire that management has to address it? I suppose that’s the point where you look for a job elsewhere. But maybe if you stuck around you would get the satisfaction of finally fixing the broken stuff (if you were still employed after the disaster that is). Wishful thinking?