4 ms·

LLMs are very useful tools for software development, but focusing on employment does not appear to really dig into if it will automate or augment labor (to use

LLMs are very useful tools for software development, but focusing on employment does not appear to really dig into if it will automate or augment labor (to use their words). Behaviors are changing not just because of outcomes but because of hype and expectations and b2b sales. You'd expect the initial corporate behaviors to look much the same whether or not LLMs turn into fully-fire-and-forget employee-replacement tools. Some nits I'd pick along those lines: >For instance, according to the most recent AI Index Report, AI systems could solve just 4.4% of coding problems on SWE-Bench, a widely used benchmark for software engineering, in 2023, but performance increased to 71.7% in 2024 (Maslej et al., 2025). Something like this should have the context of SWE-Bench not existing before November, 2023. Pre-2023 systems were flying blind with regard to what they were going to be tested with. Post-2023 systems have been created in a world where this test exists. Hard to generalize from before/after performance. > The patterns we observe in the data appear most acutely starting in late 2022, around the time of rapid proliferation of generative AI tools. This is quite early for "replacement" of software development jobs as by their own prior statement/citation the tools even a year later, when SWE-Bench was introduced, were only hitting that 4.4% task success rate. It's timing lines up more neatly with the post-COVID-bubble tech industry slowdown. Or with the start of hype about AI productivity vs actual replaced employee productivity.

eru 1y ago
Yes, even if the underlying AI stops advancing today, it will take a while for the economy to digest and adjust to the new systems. Eg a lot of the improvements in usefulness in the last few quarters came from better tooling, not necessarily better models. But with progress continuing in the models, too, it's an even more complicated affair.
- trhway 1y ago
  Offshoring was similar - i.e. companies discovered that expensive labor here can be performed inexpensively there while senior laborers/PMs here would perform the overseeing role - and we can look at it how long it took to digest it and adjust to it. While 15-20 years ago it was all the rage, today it is just an established well understood and efficiently utilized, where applicable, practice.
  - eru 1y ago
    Yes. However it wasn't just noticing the difference in wages. That had been known since forever and didn't wake a genius. Figuring out how to produce efficiently in the cheaper places and get the goods to rich markets took more smarts and experimentation. Container shipping played a big role in that, and so did modern communication and cheaper flights.
ath3nd 1y ago
> LLMs are very useful tools for software development That's an opinion many disagree with. As a matter of fact, the only limited study up to date showed that LLMs usage decrease productivity for experienced developers by roughly 19%. Let's reserve opinions and link studies. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... My anecdotal experience, for example, is that LLMs are such a negative drain on both time and quality that one has to be really early in their career to benefit from their usage.
- yakshaving_jgt 1y ago
  I’m 15 years into my career and I write Haskell every day. I’m getting a massive productivity boost from using an LLM.
  - deleted 1y ago
    [deleted]
  - black_knight 1y ago
    How do you find the quality of the Haskell code produced by LLM? Also, how do you use the LLM when coding Haskell? Generating single functions or more?
    - yakshaving_jgt 1y ago
      I'm stuck in my ways with vim/tmux/ghci etc, so I'm not using some AI IDE. I write stuff into ChatGPT and use the output, copying manually, or writing it myself with inspiration from what I get. I feed it a fair bit of context (like, say, a production module with a load of database queries, and the associated spec module) so that it copies the structure and patterns that I've established. The quality of the Haskell code is about as good as I would have written myself, though I think it falls for primitive obsession more than I would. Still, I can add those abstractions myself after the fact. Maybe one of the reasons I'm getting good results is because the LLM effectively has to argue with GHC, and GHC always wins here. I've found that it's a superpower also for finding logic bugs that I've missed, and for writing SQL queries (which I was never that good at).
hochstenbach 1y ago
One would expect that if such studies indeed indicate that AI has an effect on early-career workers in AI-exposed occupations, that this would be a global effect. I wonder if there are good comparable non-US studies available.
- moi2388 1y ago
  As a non-US citizen, in my EU country we’re still starving for new programmers.
  - trhway 1y ago
    Poland? Sometimes ago i looked up salaries in Warsaw - it were like $10-$20K/month which as i understand is pretty high by EU standards.
    - yurishimo 1y ago
      Really? That's crazy. I'm earning a bit over 5k in the Netherlands. Granted, not Amsterdam, but still.
      - trhway 1y ago
        Our guys in St.Petersburg 20 years ago were pulling $3-6K. Granted the life there comes with some risks and inconveniences :) You're probably working in a domestic company which usually pays less than offshored jobs by a large transnational (and domestics say in Russia were paying significantly less than the offshored). I don't think many companies do significant offshoring into Western Europe though.
      - wiz21c 1y ago
        before or after taxes ?
        dathinab 1y ago
        Before taxes, always before taxes. Comparing salaries between countries with vastly different approaches to taxes, health insurance, living cost, hidden side costs etc. is hard and can easily be hugely misleading. Not even including any subtle things just what "before and after taxes" means can differ. E.g. where I live "after taxes" does for most people not just deduct taxes but also the base health insurance cost and some other things (but to make it more fun, only most but not all people. This means what after tax means can differ between neighbors ). And then there are so many hidden cost which can influence taxes, e.g. in some areas you practically have to have a car this means that in effect the cost of a car isn't that different from a fixed sum tax, if you consider social standards it even scales with income up to a certain point income like most taxes (and cost x10+ if you can't drive a car for health reasons). On the other hand if you live in a area with decent public transportation and then a car is a Luxus good, but someone has to pay for the public transportation, and if you area isn't overrun by well paying tourists this means you likely pay more tax (but as public transportation tends to scale better you still likely save money especially if you aren't wealthy). Anyway so comparing after tax in vastly different countries is IMHO a folly. And even before tax is tricky but you have to choose something. I guess another option is "what to frugal living people with reasonable health insurance and rent have left over at the end of the month" is theoretically the better statistic, but just not practical.
NitpickLawyer 1y ago
> Hard to generalize from before/after performance. While this is true, there are ways to test (open models) on tasks created after the model was released. We see good numbers there as well, so something is generalising there.
ludicrousdispla 1y ago
It's funny the lengths to which companies will go in order to avoid work.
- godelski 1y ago
  It's also funny how much they'll spend to save a few pennies
- wazoox 1y ago
  Well... > The contest between the capitalist and the wage-labourer dates back to the very origin of capital. It raged on throughout the whole manufacturing period. [112] But only since the introduction of machinery has the workman fought against the instrument of labour itself, the material embodiment of capital. He revolts against this particular form of the means of production, as being the material basis of the capitalist mode of production. > [...] > The instrument of labour, when it takes the form of a machine, immediately becomes a competitor of the workman himself. [116] The self-expansion of capital by means of machinery is thenceforward directly proportional to the number of the workpeople, whose means of livelihood have been destroyed by that machinery. The whole system of capitalist production is based on the fact that the workman sells his labour-power as a commodity. Division of labour specialises this labour-power, by reducing it to skill in handling a particular tool. So soon as the handling of this tool becomes the work of a machine, then, with the use-value, the exchange-value too, of the workman’s labour-power vanishes; the workman becomes unsaleable, like paper money thrown out of currency by legal enactment. Karl Marx, The Capital, Book I Chapter 15. https://www.marxists.org/archive/marx/works/1867-c1/ch15.htm https://www.marxists.org/archive/marx/works/1867-c1/ch15.htm
dathinab 1y ago
> like this should have the context of SWE-Bench not existing before November, 2023. Given the absurdly common mal practice(1) of training LLMs on/for tests i.e. what you could describe as training on the test set any widely used/industry standard test to evaluate LLMs is not really worth half of what it claims it is. (1): Which is at least half intend, but also to some degree accident due to web scrabbling, model cross training etc. having a high chance to accidentally sneak in test data. In the end you have to have your own tests to evaluate agent/LLM performance, and worse you have to not make them public out of fare of scientific malpractice turning them worthless. Tbh. that is a pretty shitty situation.
adroniser 1y ago
This suggests people should pre-register benchmarks. Because currently it feels like there is little incentive to publish benchmarks that models saturate.

eru 1y ago

Yes, even if the underlying AI stops advancing today, it will take a while for the economy to digest and adjust to the new systems. Eg a lot of the improvements in usefulness in the last few quarters came from better tooling, not necessarily better models. But with progress continuing in the models, too, it's an even more complicated affair.

trhway 1y ago
Offshoring was similar - i.e. companies discovered that expensive labor here can be performed inexpensively there while senior laborers/PMs here would perform the overseeing role - and we can look at it how long it took to digest it and adjust to it. While 15-20 years ago it was all the rage, today it is just an established well understood and efficiently utilized, where applicable, practice.
- eru 1y ago
  Yes. However it wasn't just noticing the difference in wages. That had been known since forever and didn't wake a genius. Figuring out how to produce efficiently in the cheaper places and get the goods to rich markets took more smarts and experimentation. Container shipping played a big role in that, and so did modern communication and cheaper flights.

ath3nd 1y ago

> LLMs are very useful tools for software development That's an opinion many disagree with. As a matter of fact, the only limited study up to date showed that LLMs usage decrease productivity for experienced developers by roughly 19%. Let's reserve opinions and link studies. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... My anecdotal experience, for example, is that LLMs are such a negative drain on both time and quality that one has to be really early in their career to benefit from their usage.

yakshaving_jgt 1y ago
I’m 15 years into my career and I write Haskell every day. I’m getting a massive productivity boost from using an LLM.
- deleted 1y ago
  [deleted]
- black_knight 1y ago
  How do you find the quality of the Haskell code produced by LLM? Also, how do you use the LLM when coding Haskell? Generating single functions or more?
  - yakshaving_jgt 1y ago
    I'm stuck in my ways with vim/tmux/ghci etc, so I'm not using some AI IDE. I write stuff into ChatGPT and use the output, copying manually, or writing it myself with inspiration from what I get. I feed it a fair bit of context (like, say, a production module with a load of database queries, and the associated spec module) so that it copies the structure and patterns that I've established. The quality of the Haskell code is about as good as I would have written myself, though I think it falls for primitive obsession more than I would. Still, I can add those abstractions myself after the fact. Maybe one of the reasons I'm getting good results is because the LLM effectively has to argue with GHC, and GHC always wins here. I've found that it's a superpower also for finding logic bugs that I've missed, and for writing SQL queries (which I was never that good at).

hochstenbach 1y ago

One would expect that if such studies indeed indicate that AI has an effect on early-career workers in AI-exposed occupations, that this would be a global effect. I wonder if there are good comparable non-US studies available.

moi2388 1y ago
As a non-US citizen, in my EU country we’re still starving for new programmers.
- trhway 1y ago
  Poland? Sometimes ago i looked up salaries in Warsaw - it were like $10-$20K/month which as i understand is pretty high by EU standards.
  - yurishimo 1y ago
    Really? That's crazy. I'm earning a bit over 5k in the Netherlands. Granted, not Amsterdam, but still.
    - trhway 1y ago
      Our guys in St.Petersburg 20 years ago were pulling $3-6K. Granted the life there comes with some risks and inconveniences :) You're probably working in a domestic company which usually pays less than offshored jobs by a large transnational (and domestics say in Russia were paying significantly less than the offshored). I don't think many companies do significant offshoring into Western Europe though.
    - wiz21c 1y ago
      before or after taxes ?
      - dathinab 1y ago
        Before taxes, always before taxes. Comparing salaries between countries with vastly different approaches to taxes, health insurance, living cost, hidden side costs etc. is hard and can easily be hugely misleading. Not even including any subtle things just what "before and after taxes" means can differ. E.g. where I live "after taxes" does for most people not just deduct taxes but also the base health insurance cost and some other things (but to make it more fun, only most but not all people. This means what after tax means can differ between neighbors ). And then there are so many hidden cost which can influence taxes, e.g. in some areas you practically have to have a car this means that in effect the cost of a car isn't that different from a fixed sum tax, if you consider social standards it even scales with income up to a certain point income like most taxes (and cost x10+ if you can't drive a car for health reasons). On the other hand if you live in a area with decent public transportation and then a car is a Luxus good, but someone has to pay for the public transportation, and if you area isn't overrun by well paying tourists this means you likely pay more tax (but as public transportation tends to scale better you still likely save money especially if you aren't wealthy). Anyway so comparing after tax in vastly different countries is IMHO a folly. And even before tax is tricky but you have to choose something. I guess another option is "what to frugal living people with reasonable health insurance and rent have left over at the end of the month" is theoretically the better statistic, but just not practical.

NitpickLawyer 1y ago

> Hard to generalize from before/after performance. While this is true, there are ways to test (open models) on tasks created after the model was released. We see good numbers there as well, so something is generalising there.

ludicrousdispla 1y ago

It's funny the lengths to which companies will go in order to avoid work.

godelski 1y ago
It's also funny how much they'll spend to save a few pennies
wazoox 1y ago
Well... > The contest between the capitalist and the wage-labourer dates back to the very origin of capital. It raged on throughout the whole manufacturing period. [112] But only since the introduction of machinery has the workman fought against the instrument of labour itself, the material embodiment of capital. He revolts against this particular form of the means of production, as being the material basis of the capitalist mode of production. > [...] > The instrument of labour, when it takes the form of a machine, immediately becomes a competitor of the workman himself. [116] The self-expansion of capital by means of machinery is thenceforward directly proportional to the number of the workpeople, whose means of livelihood have been destroyed by that machinery. The whole system of capitalist production is based on the fact that the workman sells his labour-power as a commodity. Division of labour specialises this labour-power, by reducing it to skill in handling a particular tool. So soon as the handling of this tool becomes the work of a machine, then, with the use-value, the exchange-value too, of the workman’s labour-power vanishes; the workman becomes unsaleable, like paper money thrown out of currency by legal enactment. Karl Marx, The Capital, Book I Chapter 15. https://www.marxists.org/archive/marx/works/1867-c1/ch15.htm https://www.marxists.org/archive/marx/works/1867-c1/ch15.htm

dathinab 1y ago

> like this should have the context of SWE-Bench not existing before November, 2023. Given the absurdly common mal practice(1) of training LLMs on/for tests i.e. what you could describe as training on the test set any widely used/industry standard test to evaluate LLMs is not really worth half of what it claims it is. (1): Which is at least half intend, but also to some degree accident due to web scrabbling, model cross training etc. having a high chance to accidentally sneak in test data. In the end you have to have your own tests to evaluate agent/LLM performance, and worse you have to not make them public out of fare of scientific malpractice turning them worthless. Tbh. that is a pretty shitty situation.

adroniser 1y ago

This suggests people should pre-register benchmarks. Because currently it feels like there is little incentive to publish benchmarks that models saturate.