3 ms·

I was a bit disappointed by your response because, from the way you started it, I was expecting a stronger argument. I do agree with your point, but I think a k

by prmph 11mo ago

I was a bit disappointed by your response because, from the way you started it, I was expecting a stronger argument. I do agree with your point, but I think a key aspect of the false dichotomy is that there is evidence that AI is not actually "getting things done"

linsomniac 11mo ago
>there is evidence that AI is not actually "getting things done" But there is also evidence that AI is actually getting things done, right? Most of the evidence that AI can't get things done that I've seen tends to be along the lines of asking it to do a large job and it struggling. It seems like a lot of people stop there, and either don't investigate problems where it might be a better fit.
- aleph_minus_one 11mo ago
  > It seems like a lot of people stop there, and either don't investigate problems where it might be a better fit. The AI sceptics do think deeply about where AI might be a better fit. They indeed thought deeply about this, but for every hypthetical use case they could come up with, they had to conclude that - AI has to become much much more reliable to be suitable for this use case - the current AI architectures (as "the thing that bigtech markets") will likely by principle never be able to achieve this kind of reliability This is exactly why these AI-sceptical people got so sceptical about AI, and also the reason why these AI sceptics got so vocal about their opinions.
- queenkjuul 11mo ago
  I've put far more time than i should have trying to get AI to successfully complete tasks of varying sizes in our codebase at work. It simply cannot do things reliably and adequately when working in a large codebase. It lacks sufficient context, it ignores established conventions, worst of all, it often ignores instructions (endless unnecessary comments being my personal biggest peeve). So i think i have, in fact, tried my best to use it. It's great for little tiny things. Give me a one-off script to transform some command's output, translate some code from Python to Typescript, write individual unit tests for individual functions. But none of that is transforming how i do my job, it's saving me minutes, not hours. Nobody at my company is getting serious quantities of programming done with AI, either, and not for lack of trying. I've never been one to claim it's useless, just that it's usefulness (i.e. "how much is getting done") is drastically overblown.
  - linsomniac 11mo ago
    I think we're largely in agreement here, though I wouldn't go so far as to say it's limited to "little tiny" things, but I guess that's a matter of scale. I use it for a lot of tooling, which is typically in the 500-5,000 line size, and it works really well for these sorts of things. A lot of them it will just one-shot and not break a sweat. I have cases where it saves hours for sure, but they are fewer and further between. Last week we used it to solve 600+ linting warnings in 25 year old code, which probably saved me the better part of a day. It did a fantastic job of converting %-format strings to f-strings. I created a skill telling it how to test a %-to-f conversion in isolation, and it was able to use that skill to flawlessly convert all of our strings to modern usage.
- mat_b 11mo ago
  Exactly. Break the problem down and interactively plan before generating code. I'm getting a lot more done than I could have without AI by using this approach with the agent.
- lmm 11mo ago
  > But there is also evidence that AI is actually getting things done, right? Is there? I haven't seen a single AI success story that rang true, that wasn't coming from someone with a massive financial interest in it being successful. A few people subjectively think AI is making them more productive, but there's no real objective validation of that; they're not producing super-impressive products, and there was that recent study that had senior developers thinking they were being more productive using AI when in fact it was the opposite.
  - linsomniac 11mo ago
    You seem to be setting a high bar (AI success stories don't ring true), while taking the study as fact. This feels like a cognitive bias. I believe you are talking about the study: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. It is an interesting data point, but it's far from conclusive. It studied 16 developers working on large (1MLOC+) codebases, and the AI tooling struggles with large codebases (new tools like Brokk are attempting to improve performance there). The authors acknowledge that participants dropped hard AI-disallowed issues "reducing the average AI-disallowed difficulty". Some of the selected developers seem to have been inexperienced at AI use. Smaller tools and codebases and unfamiliar code are sweet spots for the AI tools. When I need a tool to help me do my job, the AIs can take a couple sentence description and turn it into working code, often on the first go. Monitoring plugins, automation tools, programs in the few thousand lines of code, writing tests, these are all things the AIs are REALLY good at. Also: asking questions about the code. A few examples: Last night I had Claude Code implement a change to the helix editor so that if you go back into a file you previously edited, it takes you back to the spot you were at in your last editing session. I don't know the helix code nor Rust at all. Claude was able to implement this in the background while I was working on another task and then watching TV in the evening. A few weeks ago I used Claude Code to fix 600+ linting errors in 20 year old code, in an evening while watching TV, these easily would have taken a day to do manually. A few months ago Claude built me a "rsync but for block devices" program; I did this one as a comparison of writing it manually vs vibe coding it with Claude, and Claude had significant advantages. But, I'm guessing these will fall into the "does not ring true" category, probably also "no real objective validation". But to me, personally, there is absolutely evidence that AI is actually getting things done.
- rsynnott 11mo ago
  There not much in the way of what you’d call strong positive evidence. Lots of user testimonials, which are, as always, kinda useless. The few serious studies attempting to measure out if (vs asking people “do you think this helped you”; again, that’s not useful evidence of anything), seem to have come come out anywhere from “a wash” to “mildly detrimental”.
brazukadev 11mo ago
Your argument is much weaker, still about filling checkboxes above all else.

by prmph 11mo ago

linsomniac 11mo ago

>there is evidence that AI is not actually "getting things done" But there is also evidence that AI is actually getting things done, right? Most of the evidence that AI can't get things done that I've seen tends to be along the lines of asking it to do a large job and it struggling. It seems like a lot of people stop there, and either don't investigate problems where it might be a better fit.

aleph_minus_one 11mo ago
> It seems like a lot of people stop there, and either don't investigate problems where it might be a better fit. The AI sceptics do think deeply about where AI might be a better fit. They indeed thought deeply about this, but for every hypthetical use case they could come up with, they had to conclude that - AI has to become much much more reliable to be suitable for this use case - the current AI architectures (as "the thing that bigtech markets") will likely by principle never be able to achieve this kind of reliability This is exactly why these AI-sceptical people got so sceptical about AI, and also the reason why these AI sceptics got so vocal about their opinions.
queenkjuul 11mo ago
I've put far more time than i should have trying to get AI to successfully complete tasks of varying sizes in our codebase at work. It simply cannot do things reliably and adequately when working in a large codebase. It lacks sufficient context, it ignores established conventions, worst of all, it often ignores instructions (endless unnecessary comments being my personal biggest peeve). So i think i have, in fact, tried my best to use it. It's great for little tiny things. Give me a one-off script to transform some command's output, translate some code from Python to Typescript, write individual unit tests for individual functions. But none of that is transforming how i do my job, it's saving me minutes, not hours. Nobody at my company is getting serious quantities of programming done with AI, either, and not for lack of trying. I've never been one to claim it's useless, just that it's usefulness (i.e. "how much is getting done") is drastically overblown.
- linsomniac 11mo ago
  I think we're largely in agreement here, though I wouldn't go so far as to say it's limited to "little tiny" things, but I guess that's a matter of scale. I use it for a lot of tooling, which is typically in the 500-5,000 line size, and it works really well for these sorts of things. A lot of them it will just one-shot and not break a sweat. I have cases where it saves hours for sure, but they are fewer and further between. Last week we used it to solve 600+ linting warnings in 25 year old code, which probably saved me the better part of a day. It did a fantastic job of converting %-format strings to f-strings. I created a skill telling it how to test a %-to-f conversion in isolation, and it was able to use that skill to flawlessly convert all of our strings to modern usage.
mat_b 11mo ago
Exactly. Break the problem down and interactively plan before generating code. I'm getting a lot more done than I could have without AI by using this approach with the agent.
lmm 11mo ago
> But there is also evidence that AI is actually getting things done, right? Is there? I haven't seen a single AI success story that rang true, that wasn't coming from someone with a massive financial interest in it being successful. A few people subjectively think AI is making them more productive, but there's no real objective validation of that; they're not producing super-impressive products, and there was that recent study that had senior developers thinking they were being more productive using AI when in fact it was the opposite.
- linsomniac 11mo ago
  You seem to be setting a high bar (AI success stories don't ring true), while taking the study as fact. This feels like a cognitive bias. I believe you are talking about the study: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. It is an interesting data point, but it's far from conclusive. It studied 16 developers working on large (1MLOC+) codebases, and the AI tooling struggles with large codebases (new tools like Brokk are attempting to improve performance there). The authors acknowledge that participants dropped hard AI-disallowed issues "reducing the average AI-disallowed difficulty". Some of the selected developers seem to have been inexperienced at AI use. Smaller tools and codebases and unfamiliar code are sweet spots for the AI tools. When I need a tool to help me do my job, the AIs can take a couple sentence description and turn it into working code, often on the first go. Monitoring plugins, automation tools, programs in the few thousand lines of code, writing tests, these are all things the AIs are REALLY good at. Also: asking questions about the code. A few examples: Last night I had Claude Code implement a change to the helix editor so that if you go back into a file you previously edited, it takes you back to the spot you were at in your last editing session. I don't know the helix code nor Rust at all. Claude was able to implement this in the background while I was working on another task and then watching TV in the evening. A few weeks ago I used Claude Code to fix 600+ linting errors in 20 year old code, in an evening while watching TV, these easily would have taken a day to do manually. A few months ago Claude built me a "rsync but for block devices" program; I did this one as a comparison of writing it manually vs vibe coding it with Claude, and Claude had significant advantages. But, I'm guessing these will fall into the "does not ring true" category, probably also "no real objective validation". But to me, personally, there is absolutely evidence that AI is actually getting things done.
rsynnott 11mo ago
There not much in the way of what you’d call strong positive evidence. Lots of user testimonials, which are, as always, kinda useless. The few serious studies attempting to measure out if (vs asking people “do you think this helped you”; again, that’s not useful evidence of anything), seem to have come come out anywhere from “a wash” to “mildly detrimental”.

brazukadev 11mo ago

Your argument is much weaker, still about filling checkboxes above all else.