8 ms·

So, a couple years ago Microsoft was the first large, public-facing software organization to make LLM-assisted coding a big part of their production. If LLM's

So, a couple years ago Microsoft was the first large, public-facing software organization to make LLM-assisted coding a big part of their production. If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now. So, either LLM-assisted coding is not delivering the benefits some thought it would, or Microsoft, despite being an early investor in OpenAI, is not using it much internally on things that really matter to them (like Windows). Either way, I'm not impressed.

pawelduda 9mo ago
If they used copilot and it was years ago, I'm actually impressed there are no reports of Windows PC's exploding
Octoth0rpe 9mo ago
> If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now. That productivity may not be visible. I think MS's move-everything-to-rust initiate would be one hell of an endorsement if they manage to make visible progress on that in the next couple of years.
- Someone1234 9mo ago
  Microsoft has no "move-everything-to-Rust initiative" and never did. That was a bunch of clickbait created based on the personal comments by a single Microsoft developer.
  - Octoth0rpe 9mo ago
    Thanks for the heads up, I was not following closely.
- mattgreenrocks 9mo ago
  > That productivity may not be visible. I'm not sure what your take is, but this reads like goalpost shifting. If one of the biggest orgs that practically mandates some amount of LLM use cannot surface productivity gains from them after using them for several years, then that speaks volumes. Reality has a way of showing itself eventually.
Someone1234 9mo ago
I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped. Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one, trying to maximize short-term shareholder value at the cost of long-term company reputation/growth. It is very common and typical of US Corporate culture today, and catastrophic in the long-run. [0] https://arstechnica.com/information-technology/2014/08/how-microsoft-dragged-its-development-practices-into-the-21st-century/ https://arstechnica.com/information-technology/2014/08/how-m... [1] https://www.reuters.com/article/business/microsoft-expected-to-announce-job-cuts-this-week-bloomberg-idUSL4N0PQ12I/ https://www.reuters.com/article/business/microsoft-expected-...
- Night_Thastus 9mo ago
  Microsoft fired their QA because at the end of the day, they are beholden to shareholders. And those shareholders want higher profits. And if you want higher profits, you cut costs. It's not a culture problem. It's a 'being a business' problem, which unfortunately affects all publicly-traded companies.
  - halapro 9mo ago
    Are businesses expected to boom and bust? Cost cutting is fine if you don't kill the company in the process.
    - Night_Thastus 9mo ago
      They know MS isn't going anywhere. Windows is too entrenched, users don't care or have feasible alternatives, for a variety of reasons. Plus, MS isn't in the OS business. They're in the data/metrics business.
      - zx8080 9mo ago
        > Plus, MS isn't in the OS business. They're in the data/metrics business. Datadog is. And snowflake. Even Google is. But MS does not like it's centered around data/metrics.
adamrezich 9mo ago
Imagine a world where Microsoft was pushing “Copilot” integration everywhere, just as they are in this one—but the proof was, actually, in the pudding. Windows was categorically improving, without regression, with each subsequent update. Long-standing frustrations with the operating system experience were gradually being ironed out. Parts of the system that were slow, frustrating, convoluted, or all three, were being thoughtfully redesigned without breaking backwards compatibility, and we were watching this all unfold in real time, in awe of the power of “AI”, eyes wide with hope for the future of software, and computing in general. Think of how dramatically this hypothetical alternate reality differs from the one we live in, and then consider just how galling it is that these people have the nerve to piss on our leg and then tell us it's raining. Things are not getting better. This supposedly-magical new technology isn't observably improving things where it matters most—rather, it's demonstrably hastening the decline of the baseline day-to-day software that we depend upon.
- rossdavidh 9mo ago
  Interesting thought experiment. In that alternate reality, their shareholders would probably be shouting "why would you give competitors access to this awesome tool?!"
- heliumtera 9mo ago
  But web people can write css faster so I think it is a net positive?
  - alex1138 9mo ago
    Yeah, but I'm very worried about subtle errors getting introduced
- Telaneo 9mo ago
  The distance between the promise and the reality really is huge. On some level I wished they'd just promise less, because it's not like LLMs compleatly useless. I don't find much use in them, but some clearly do. They do them. But since the entire economy has apparently bet the farm on AI, underpromising isn't really an option, while underdelivering is a problem for future Microslop and co.
rich_sasha 9mo ago
They weren't great before LLMs either. Also, it seems from the outside like a dysfunctional organisation, or at least with incentives heavily misaligned with their users. Replace LLMs with a bunch of 10x engineers and it will still be bad in an environment like this. So not sure how much to blame the LLMs - or in fact how much MS is really using them. Poor souls have to use MS AI tools, I almost feel sorry for them.
- BizarroLand 9mo ago
  They hit peak with Windows 7 and will never have an operating system that good again. Some flavors of Linux are approaching the Windows 7 peak as well as far as ease of use for newbies, software "just working", and for familiarity for users of other OS's. Their days as the default OS for most people are numbered unless they pull an incredible heel turn.
  - troyvit 9mo ago
    On a whim I gave my 14 year old an old System76 laptop with ElementaryOS on it then sent her back to her Mom's house on the other end of the world. Then she switched schools and ended up requiring a laptop instead of an iPad to do her work. I about crapped my pants but she's been using that laptop almost problem-free for two months now (two glitches with Firefox that she got around). She even figured out how to install Sober so she can play Roblox. While that probably says as much about my parenting as Linux's progress I have to say, I'm pretty impressed.
  - charcircuit 9mo ago
    Have you tried Windows 11? The WSL2 integration works really well. And the work that is being done in regards to safe vms so games can move away from kernel anticheat is also exciting.
    - pjmlp 9mo ago
      As someone that was really into WinRT, pity that the whole UWP stack went bust, it was so mismanaged that outside Windows team themselves no one else cares any longer.
j1elo 9mo ago
It's not LLMs. It's returns-driven-development.
- heliumtera 9mo ago
  But the second was always the case, windows and everything else is getting shittier so fast it would require a prompt explanation if we didn't have one.
  - prmoustache 9mo ago
    Rose tinted glasses. Windows 2000 may have been bearable but windows has always been shitty.
- ljm 9mo ago
  Growth at any cost. Once growth is unable to increase the wealth of the shareholders the money has to be diverted from elsewhere, via cuts. Money gotta keep flowing upwards.
SloppyDrive 9mo ago
I fully believe highly skilled people can get a great benefit from LLM tools; probably not 10x; but enough that its noticeable. The key thing for me is that it only works when the LLM is used for tasks below the devs skill level; It can speed up somebody good, but it also makes the output of low-skill devs much harder to deal with. The issues are more subtle, the volume is greater, and there is no human reasoning chain to follow when debugging. So you combine that with a company that has staff in low skill regions, and uses outsourcing, and while there might be some high skill teams that got a speed up, the org is structured in a way that its irrelevant.
- claysmithr 9mo ago
  I think they keyword is "highly skilled." However, not everyone using the LLM will be highly skilled, especially juniors new to the industry.
serf 9mo ago
this reasoning is flawed. wouldn't a for-profit company just balance the workforce for the productivity gained to increase overall profit? some person is 10x 'more productive' (whatever that means) , let's cut 9 jobs. Although to your grander point, employment during the LLM-embrace period seems fairly stable.[0] [0]: https://stockanalysis.com/stocks/msft/employees/ https://stockanalysis.com/stocks/msft/employees/
baka367 9mo ago
Oh it did help. Microsoft went all in on do more with less and fired/reorged significant part of the company. Wouldn’t be surprised if the outage is caused by new team taking something over with near zero documentation while all the tribal knowledge was torched away
njhnjhnjhnjh 9mo ago
[dead]
rk06 9mo ago
Microsoft is not even using dotnet core and what not, internally. SLT is very hard on adopting AI, but not much on getting results
steve1977 9mo ago
If anything, we see a decrease, not an increase.
Someone 9mo ago
Or LLMs weren’t good enough yet years ago, but the growth curve looked so promising that an investment seemed a good idea. Also: do you have a reference for “a couple years ago Microsoft [made] LLM-assisted coding a big part of their production”? I know they started investing, mentioning future benefits, but don’t remember them saying their Windows development team (heavily) relying on it.
dzonga 9mo ago
exactly my thoughts as well - if LLM really were massive productive booms - then we would see the number of bugs in major software platforms going down - we would see more features - but neither is happening so yeah we're being sold a bag of air
gchamonlive 9mo ago
I think it's naive to believe AI is used primarily for productivity boost. It's used mainly for cost reduction and to increase profits, even if quality and productivity take a hit in the process.
felixgallo 9mo ago
(1) a couple of years ago, LLMs for coding sucked pretty bad. (2) LLMs are a force multiplier. If you start with a negative number, then your coefficient makes things worse. (3) Microsoft has never been a place of quality. It's not organized for that, it doesn't have that as its philosophy, and so you should never be surprised that it doesn't deliver that.
bodge5000 9mo ago
The argument I usually hear is that you only truly get the 10x improvements with <new-model> (right now, Opus 4.5), so they've only had a few months, not years. In a few months, it'll turn out that <new-model> wasn't actually capable of that, but <new-new-model> is, and as that's not been out long its unfair to judge so early. And so the cycle begins anew
- rossdavidh 9mo ago
  Very true. Look at the wikipedia article for "AI Winter" to see how old this cycle truly is...
MrBuddyCasino 9mo ago
Everyone should read "The Bear Case for AI" thread: "The bear case for AI is that bringing 10x or 100x or 1000x more intelligence to America will not change anything because U.S. institutions are already designed to ignore or waste intelligence and have no idea what to do with any more of it." https://twitter.com/mmjukic/status/2014255931215716545 https://twitter.com/mmjukic/status/2014255931215716545
austin-cheney 9mo ago
Its a misunderstanding of costs. Its the same misunderstanding of thermodynamics applied to dieting. Inputs do not equal outputs, or even scaled outputs. In any dynamic system there are costs to storage and costs to processing. This is how you can increase your calorie intake and still rapidly lose weight, yes, I have lost 30 pounds that way. In the case of LLMs the only way LLM use becomes profitable is if this condition are achieved: processing + storage + input + validation + maintenance < manual output + manual support If you want to see a 10x savings then multiply the cost of manual output by 10. While LLM profit is achievable in some scenarios a t0x improvement in most scenarios is highly improbable.
isk517 9mo ago
People are pointing their fingers at QA, and while that is a big part of it I think the bigger issue is, like you said, them not really caring about some of their core products. Windows 11 seems to exist purely so that they can earn passive ad revenue while vacuuming up user data, Office 365 is now just a pile of mature applications that are slowly getting worse and new applications that are too unfinished to be actively useful.
ManlyBread 9mo ago
I would like to point out that not even a week ago Satya Nadella stated that someone should finally do something really useful with AI because if no one does then they'll lose the social permission to burn all the energy on training and running the models: https://www.pcgamer.com/software/ai/microsoft-ceo-warns-that-we-must-do-something-useful-with-ai-or-theyll-lose-social-permission-to-burn-electricity-on-it/ https://www.pcgamer.com/software/ai/microsoft-ceo-warns-that... Mr. Nadella, why not lead by example and make Windows the most amazing operating system ever created with the help of Copilot? What's the holdup?

pawelduda 9mo ago

If they used copilot and it was years ago, I'm actually impressed there are no reports of Windows PC's exploding

Octoth0rpe 9mo ago

> If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now. That productivity may not be visible. I think MS's move-everything-to-rust initiate would be one hell of an endorsement if they manage to make visible progress on that in the next couple of years.

Someone1234 9mo ago
Microsoft has no "move-everything-to-Rust initiative" and never did. That was a bunch of clickbait created based on the personal comments by a single Microsoft developer.
- Octoth0rpe 9mo ago
  Thanks for the heads up, I was not following closely.
mattgreenrocks 9mo ago
> That productivity may not be visible. I'm not sure what your take is, but this reads like goalpost shifting. If one of the biggest orgs that practically mandates some amount of LLM use cannot surface productivity gains from them after using them for several years, then that speaks volumes. Reality has a way of showing itself eventually.

Someone1234 9mo ago

I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped. Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one, trying to maximize short-term shareholder value at the cost of long-term company reputation/growth. It is very common and typical of US Corporate culture today, and catastrophic in the long-run. [0] https://arstechnica.com/information-technology/2014/08/how-microsoft-dragged-its-development-practices-into-the-21st-century/ https://arstechnica.com/information-technology/2014/08/how-m... [1] https://www.reuters.com/article/business/microsoft-expected-to-announce-job-cuts-this-week-bloomberg-idUSL4N0PQ12I/ https://www.reuters.com/article/business/microsoft-expected-...

Night_Thastus 9mo ago
Microsoft fired their QA because at the end of the day, they are beholden to shareholders. And those shareholders want higher profits. And if you want higher profits, you cut costs. It's not a culture problem. It's a 'being a business' problem, which unfortunately affects all publicly-traded companies.
- halapro 9mo ago
  Are businesses expected to boom and bust? Cost cutting is fine if you don't kill the company in the process.
  - Night_Thastus 9mo ago
    They know MS isn't going anywhere. Windows is too entrenched, users don't care or have feasible alternatives, for a variety of reasons. Plus, MS isn't in the OS business. They're in the data/metrics business.
    - zx8080 9mo ago
      > Plus, MS isn't in the OS business. They're in the data/metrics business. Datadog is. And snowflake. Even Google is. But MS does not like it's centered around data/metrics.

adamrezich 9mo ago

Imagine a world where Microsoft was pushing “Copilot” integration everywhere, just as they are in this one—but the proof was, actually, in the pudding. Windows was categorically improving, without regression, with each subsequent update. Long-standing frustrations with the operating system experience were gradually being ironed out. Parts of the system that were slow, frustrating, convoluted, or all three, were being thoughtfully redesigned without breaking backwards compatibility, and we were watching this all unfold in real time, in awe of the power of “AI”, eyes wide with hope for the future of software, and computing in general. Think of how dramatically this hypothetical alternate reality differs from the one we live in, and then consider just how galling it is that these people have the nerve to piss on our leg and then tell us it's raining. Things are not getting better. This supposedly-magical new technology isn't observably improving things where it matters most—rather, it's demonstrably hastening the decline of the baseline day-to-day software that we depend upon.

rossdavidh 9mo ago
Interesting thought experiment. In that alternate reality, their shareholders would probably be shouting "why would you give competitors access to this awesome tool?!"
heliumtera 9mo ago
But web people can write css faster so I think it is a net positive?
- alex1138 9mo ago
  Yeah, but I'm very worried about subtle errors getting introduced
Telaneo 9mo ago
The distance between the promise and the reality really is huge. On some level I wished they'd just promise less, because it's not like LLMs compleatly useless. I don't find much use in them, but some clearly do. They do them. But since the entire economy has apparently bet the farm on AI, underpromising isn't really an option, while underdelivering is a problem for future Microslop and co.

rich_sasha 9mo ago

They weren't great before LLMs either. Also, it seems from the outside like a dysfunctional organisation, or at least with incentives heavily misaligned with their users. Replace LLMs with a bunch of 10x engineers and it will still be bad in an environment like this. So not sure how much to blame the LLMs - or in fact how much MS is really using them. Poor souls have to use MS AI tools, I almost feel sorry for them.

BizarroLand 9mo ago
They hit peak with Windows 7 and will never have an operating system that good again. Some flavors of Linux are approaching the Windows 7 peak as well as far as ease of use for newbies, software "just working", and for familiarity for users of other OS's. Their days as the default OS for most people are numbered unless they pull an incredible heel turn.
- troyvit 9mo ago
  On a whim I gave my 14 year old an old System76 laptop with ElementaryOS on it then sent her back to her Mom's house on the other end of the world. Then she switched schools and ended up requiring a laptop instead of an iPad to do her work. I about crapped my pants but she's been using that laptop almost problem-free for two months now (two glitches with Firefox that she got around). She even figured out how to install Sober so she can play Roblox. While that probably says as much about my parenting as Linux's progress I have to say, I'm pretty impressed.
- charcircuit 9mo ago
  Have you tried Windows 11? The WSL2 integration works really well. And the work that is being done in regards to safe vms so games can move away from kernel anticheat is also exciting.
  - pjmlp 9mo ago
    As someone that was really into WinRT, pity that the whole UWP stack went bust, it was so mismanaged that outside Windows team themselves no one else cares any longer.

j1elo 9mo ago

It's not LLMs. It's returns-driven-development.

heliumtera 9mo ago
But the second was always the case, windows and everything else is getting shittier so fast it would require a prompt explanation if we didn't have one.
- prmoustache 9mo ago
  Rose tinted glasses. Windows 2000 may have been bearable but windows has always been shitty.
ljm 9mo ago
Growth at any cost. Once growth is unable to increase the wealth of the shareholders the money has to be diverted from elsewhere, via cuts. Money gotta keep flowing upwards.

SloppyDrive 9mo ago

I fully believe highly skilled people can get a great benefit from LLM tools; probably not 10x; but enough that its noticeable. The key thing for me is that it only works when the LLM is used for tasks below the devs skill level; It can speed up somebody good, but it also makes the output of low-skill devs much harder to deal with. The issues are more subtle, the volume is greater, and there is no human reasoning chain to follow when debugging. So you combine that with a company that has staff in low skill regions, and uses outsourcing, and while there might be some high skill teams that got a speed up, the org is structured in a way that its irrelevant.

claysmithr 9mo ago
I think they keyword is "highly skilled." However, not everyone using the LLM will be highly skilled, especially juniors new to the industry.

serf 9mo ago

this reasoning is flawed. wouldn't a for-profit company just balance the workforce for the productivity gained to increase overall profit? some person is 10x 'more productive' (whatever that means) , let's cut 9 jobs. Although to your grander point, employment during the LLM-embrace period seems fairly stable.[0] [0]: https://stockanalysis.com/stocks/msft/employees/ https://stockanalysis.com/stocks/msft/employees/

baka367 9mo ago

Oh it did help. Microsoft went all in on do more with less and fired/reorged significant part of the company. Wouldn’t be surprised if the outage is caused by new team taking something over with near zero documentation while all the tribal knowledge was torched away

njhnjhnjhnjh 9mo ago

[dead]

rk06 9mo ago

Microsoft is not even using dotnet core and what not, internally. SLT is very hard on adopting AI, but not much on getting results

steve1977 9mo ago

If anything, we see a decrease, not an increase.

Someone 9mo ago

Or LLMs weren’t good enough yet years ago, but the growth curve looked so promising that an investment seemed a good idea. Also: do you have a reference for “a couple years ago Microsoft [made] LLM-assisted coding a big part of their production”? I know they started investing, mentioning future benefits, but don’t remember them saying their Windows development team (heavily) relying on it.

dzonga 9mo ago

exactly my thoughts as well - if LLM really were massive productive booms - then we would see the number of bugs in major software platforms going down - we would see more features - but neither is happening so yeah we're being sold a bag of air

gchamonlive 9mo ago

I think it's naive to believe AI is used primarily for productivity boost. It's used mainly for cost reduction and to increase profits, even if quality and productivity take a hit in the process.

felixgallo 9mo ago

(1) a couple of years ago, LLMs for coding sucked pretty bad. (2) LLMs are a force multiplier. If you start with a negative number, then your coefficient makes things worse. (3) Microsoft has never been a place of quality. It's not organized for that, it doesn't have that as its philosophy, and so you should never be surprised that it doesn't deliver that.

bodge5000 9mo ago

The argument I usually hear is that you only truly get the 10x improvements with <new-model> (right now, Opus 4.5), so they've only had a few months, not years. In a few months, it'll turn out that <new-model> wasn't actually capable of that, but <new-new-model> is, and as that's not been out long its unfair to judge so early. And so the cycle begins anew

rossdavidh 9mo ago
Very true. Look at the wikipedia article for "AI Winter" to see how old this cycle truly is...

MrBuddyCasino 9mo ago

Everyone should read "The Bear Case for AI" thread: "The bear case for AI is that bringing 10x or 100x or 1000x more intelligence to America will not change anything because U.S. institutions are already designed to ignore or waste intelligence and have no idea what to do with any more of it." https://twitter.com/mmjukic/status/2014255931215716545 https://twitter.com/mmjukic/status/2014255931215716545

austin-cheney 9mo ago

Its a misunderstanding of costs. Its the same misunderstanding of thermodynamics applied to dieting. Inputs do not equal outputs, or even scaled outputs. In any dynamic system there are costs to storage and costs to processing. This is how you can increase your calorie intake and still rapidly lose weight, yes, I have lost 30 pounds that way. In the case of LLMs the only way LLM use becomes profitable is if this condition are achieved: processing + storage + input + validation + maintenance < manual output + manual support If you want to see a 10x savings then multiply the cost of manual output by 10. While LLM profit is achievable in some scenarios a t0x improvement in most scenarios is highly improbable.

isk517 9mo ago

People are pointing their fingers at QA, and while that is a big part of it I think the bigger issue is, like you said, them not really caring about some of their core products. Windows 11 seems to exist purely so that they can earn passive ad revenue while vacuuming up user data, Office 365 is now just a pile of mature applications that are slowly getting worse and new applications that are too unfinished to be actively useful.

ManlyBread 9mo ago

I would like to point out that not even a week ago Satya Nadella stated that someone should finally do something really useful with AI because if no one does then they'll lose the social permission to burn all the energy on training and running the models: https://www.pcgamer.com/software/ai/microsoft-ceo-warns-that-we-must-do-something-useful-with-ai-or-theyll-lose-social-permission-to-burn-electricity-on-it/ https://www.pcgamer.com/software/ai/microsoft-ceo-warns-that... Mr. Nadella, why not lead by example and make Windows the most amazing operating system ever created with the help of Copilot? What's the holdup?