5 ms·

> Look for the ways that AI works, and it can be a powerful tool. Try and figure out where it still fails, and you will see nothing but hype and hot air. Not ev

> Look for the ways that AI works, and it can be a powerful tool. Try and figure out where it still fails, and you will see nothing but hype and hot air. Not every use case is like this, but there are many. The problem is that I feel I am constantly being bombarded by people bullish on AI saying "look how great this is" but when I try to do the exact same things they are doing, it doesn't work very well for me Of course I am skeptical of positive claims as a result.

mattigames 2y ago
Exactly, thanks to all the money involved in such hype the incentives will always skew towards over spamming naive optimism about it's features.
MostlyStable 2y ago
I don't know what you are doing or why it's failed. Maybe my primary use cases really are in the top whatever percentile for AI usefulness, but it doesn't feel like it. All I know is that frontier models have already been good enough for more than a year to increase my productivity by a fair bit.
- nonchalantsui 2y ago
  Your use case is in fact in the top whatever percentile for AI usefulness. Short simple scripting that won't have to be relied on due to never being widely deployed. No large codebase it has to comb through, no need for thorough maintenance and update management, no need for efficient (and potentially rare) solutions. The only use case that would beat yours is the type of office worker that cannot write professional sounding emails but has to send them out regularly manually.
  - MostlyStable 2y ago
    I fully believe it's far better at the kind of coding/scripting that I do than the kind that real SWEs do. If for no other reason than the coding itself that I do is far far simpler and easier, so of course it's going to do better at it. However, I don't really believe that coding is the only use case. I think that there are a whole universe of other use cases that probably also get a lot of value from LLMs. I think that HN has a lot of people who are working on large software projects that are incredibly complex and have a huge numbers of interdependencies etc., and LLMs aren't quite to the point that they can very usefully contribute to that except around the edges. But I don't think that generalizing from that failure is very useful either. Most things humans do aren't that hard. There is a reason that SWE is one of the best paid jobs in the country.
    - mattmanser 2y ago
      Even a 1 month project with one good senior engineer working on it will get 20+ different files and 5,000+ loc. Real programming is on a totally different scale than what you're describing. I think that's true for most jobs. Superficially an AI looks like it can do good. But LLMs: 1. Hallucinate all the time. If they were human we'd call them compulsive liars 2. They are consistenly inconsistent, so are useless for automation 3. Are only good at anything they can copy from their data set. They can't create, only regurgitate other people's work 4. AI influencing hasn't happened yet, but will very soon start making AI LLMs useless, much like SEO has ruined search. You can bet there are a load of people already seeding the internet with a load of advertising and misinformation aimed solely at AIs and AI reinforcement
      - acuozzo 2y ago
        > Real programming is on a totally different scale than what you're describing. How "real" are we talking? When I think of "real programming" I think of flight control software for commercial airplanes and, I can assure you, 1 month != 5,000 LoC in that space.
      - jjani 2y ago
        > Even a 1 month project with one good senior engineer working on it will get 20+ different files and 5,000+ loc. For what it's worth, I mostly work on projects in the 100-200 files range, at 20-40k LoC. When using proper tooling with appropriate models, it boosts my productivity by at least 2x (being conservative). I've experimented with this by going a few days without using them, then using them again. Definitely far from the massive codebases many on here work on, small beans by HN standards. But also decidedly not just writing one-off scripts.
    - throwaway290 2y ago
      It's not about the size it's more about if the task is trivial.
  - kerkeslager 2y ago
    And... I know people who now use AI to write their professional-sounding emails, and they often don't sound as professional as they think they do. It can be easy to just skim what an AI generates and think it's okay to send if you aren't careful, but the people you send those emails to actually have to read what was written and attempt to understand it, and doing that makes you notice things that a brief skim doesn't catch. It's actually extremely irritating that I'm only half talking to the person when I email with these people.
    - skydhash 2y ago
      It's kinda like machine translated novels. You have to really be passionate about the novel to endure these kinds of translations. That's when you realize how much work novel translators do to get a coherent result.
      - Ekaros 2y ago
        Especially jarring when you have read translation that put thought in them. Noticed this in Xianxia so Chinese power-fantasy. Where selection of what to translate and what to transliterate can have huge impact. And then editorial work also becomes important if something in past need to be changed based on future information.
turtletontine 2y ago
I literally had a developer of an open source package I’m working with tell me “yeah that’s a known problem, I gave up on trying to fix it. You should just ask ChatGPT to fix it, I bet it will immediately know the answer.” Annoying response of course. But I’d never used an LLM to debug before, so I figured I’d give it a try. First: it regurgitated a bunch of documentation and basic debugging tips, which might have actually been helpful if I had just encountered this problem and had put no thought into debugging it yet. In reality, I had already spent hours on the problem. So not helpful Second: I provided some further info on environment variables I thought might be the problem. It latched on to that. “Yes that’s your problem! These environment variables are (causing the problem) because (reasons that don’t make sense). Delete them and that should fix things.” I deleted them. It changed nothing. Third: It hallucinated a magic numpy function that would solve my problem. I informed it this function did not exist, and it wrote me a flowery apology. Clearly AI coding works great for some people, but this was purely an infuriating distraction. Not only did it not solve my problem, it wasted my time and energy, and threw tons of useless and irrelevant information at me. Bad experience.
- ajkdhcb2 2y ago
  My experiences have all been like this too. I am puzzled by how some people say it works for them
  - simonw 2y ago
    I wrote this article precisely for people who are having trouble getting good results out of LLMs for coding: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
    - codegangsta 2y ago
      Just wanted to chime in and say how appreciative I’ve been about all your replies here, and overall content on AI. Your takes are super reasonable and well thought out.
    - 2y ago
kerkeslager 2y ago
It's worse than that. I see people say, "Look how great this is," and show me an example, and the example they show me is just not great. We're literally looking at the same thing, and they're excited that this LLM can do a college grads's job to the level of a third grader, and I'm just not excited about that.
kristopolous 2y ago
They're sophisticated tools at much as any other software. About 2 weeks ago I started on a streaming markdown parser for the terminal because none really existed. I've switched to human coding now but the first version was basically all llm prompting and a bunch of the code is still llm generated (maybe 80%). It's a parser, those are hard. There's stacks, states, lookaheads, look behinds, feature flags, color spaces, support for things like links and syntax highlighting... all forward streaming. Not easy https://github.com/kristopolous/Streamdown https://github.com/kristopolous/Streamdown
thisissomething 2y ago
What changed my point of view regarding LLMs was when I realized how crucial context is in increasing output quality. Treat the AI as a freelancer working on your project. How would you ask a freelancer to create a Kanban system for you? By simply asking "Create a Kanban system", or by providing them a 2-3 pages document describing features, guidelines, restrictions, requirements, dependencies, design ethos, etc? Which approach will get you closer to your objective? The same applies to LLM (when it comes to code generation). When well instructed, it can quickly generate a lot of working code, and apply the necessary fixes/changes you request inside that same context window. It still can't generate senior-level code, but it saves hours when doing grunt work or prototyping ideas. "Oh, but the code isn't perfect". Nor is the code of the average jr dev, but their codes still make it to production in thousands of companies around the world.
- joquarky 2y ago
  I see it as a knowledge multiplier. You still need to know enough about the subject to verify the output.

mattigames 2y ago

Exactly, thanks to all the money involved in such hype the incentives will always skew towards over spamming naive optimism about it's features.

MostlyStable 2y ago

I don't know what you are doing or why it's failed. Maybe my primary use cases really are in the top whatever percentile for AI usefulness, but it doesn't feel like it. All I know is that frontier models have already been good enough for more than a year to increase my productivity by a fair bit.

nonchalantsui 2y ago
Your use case is in fact in the top whatever percentile for AI usefulness. Short simple scripting that won't have to be relied on due to never being widely deployed. No large codebase it has to comb through, no need for thorough maintenance and update management, no need for efficient (and potentially rare) solutions. The only use case that would beat yours is the type of office worker that cannot write professional sounding emails but has to send them out regularly manually.
- MostlyStable 2y ago
  I fully believe it's far better at the kind of coding/scripting that I do than the kind that real SWEs do. If for no other reason than the coding itself that I do is far far simpler and easier, so of course it's going to do better at it. However, I don't really believe that coding is the only use case. I think that there are a whole universe of other use cases that probably also get a lot of value from LLMs. I think that HN has a lot of people who are working on large software projects that are incredibly complex and have a huge numbers of interdependencies etc., and LLMs aren't quite to the point that they can very usefully contribute to that except around the edges. But I don't think that generalizing from that failure is very useful either. Most things humans do aren't that hard. There is a reason that SWE is one of the best paid jobs in the country.
  - mattmanser 2y ago
    Even a 1 month project with one good senior engineer working on it will get 20+ different files and 5,000+ loc. Real programming is on a totally different scale than what you're describing. I think that's true for most jobs. Superficially an AI looks like it can do good. But LLMs: 1. Hallucinate all the time. If they were human we'd call them compulsive liars 2. They are consistenly inconsistent, so are useless for automation 3. Are only good at anything they can copy from their data set. They can't create, only regurgitate other people's work 4. AI influencing hasn't happened yet, but will very soon start making AI LLMs useless, much like SEO has ruined search. You can bet there are a load of people already seeding the internet with a load of advertising and misinformation aimed solely at AIs and AI reinforcement
    - acuozzo 2y ago
      > Real programming is on a totally different scale than what you're describing. How "real" are we talking? When I think of "real programming" I think of flight control software for commercial airplanes and, I can assure you, 1 month != 5,000 LoC in that space.
    - jjani 2y ago
      > Even a 1 month project with one good senior engineer working on it will get 20+ different files and 5,000+ loc. For what it's worth, I mostly work on projects in the 100-200 files range, at 20-40k LoC. When using proper tooling with appropriate models, it boosts my productivity by at least 2x (being conservative). I've experimented with this by going a few days without using them, then using them again. Definitely far from the massive codebases many on here work on, small beans by HN standards. But also decidedly not just writing one-off scripts.
  - throwaway290 2y ago
    It's not about the size it's more about if the task is trivial.
- kerkeslager 2y ago
  And... I know people who now use AI to write their professional-sounding emails, and they often don't sound as professional as they think they do. It can be easy to just skim what an AI generates and think it's okay to send if you aren't careful, but the people you send those emails to actually have to read what was written and attempt to understand it, and doing that makes you notice things that a brief skim doesn't catch. It's actually extremely irritating that I'm only half talking to the person when I email with these people.
  - skydhash 2y ago
    It's kinda like machine translated novels. You have to really be passionate about the novel to endure these kinds of translations. That's when you realize how much work novel translators do to get a coherent result.
    - Ekaros 2y ago
      Especially jarring when you have read translation that put thought in them. Noticed this in Xianxia so Chinese power-fantasy. Where selection of what to translate and what to transliterate can have huge impact. And then editorial work also becomes important if something in past need to be changed based on future information.

turtletontine 2y ago

I literally had a developer of an open source package I’m working with tell me “yeah that’s a known problem, I gave up on trying to fix it. You should just ask ChatGPT to fix it, I bet it will immediately know the answer.” Annoying response of course. But I’d never used an LLM to debug before, so I figured I’d give it a try. First: it regurgitated a bunch of documentation and basic debugging tips, which might have actually been helpful if I had just encountered this problem and had put no thought into debugging it yet. In reality, I had already spent hours on the problem. So not helpful Second: I provided some further info on environment variables I thought might be the problem. It latched on to that. “Yes that’s your problem! These environment variables are (causing the problem) because (reasons that don’t make sense). Delete them and that should fix things.” I deleted them. It changed nothing. Third: It hallucinated a magic numpy function that would solve my problem. I informed it this function did not exist, and it wrote me a flowery apology. Clearly AI coding works great for some people, but this was purely an infuriating distraction. Not only did it not solve my problem, it wasted my time and energy, and threw tons of useless and irrelevant information at me. Bad experience.

ajkdhcb2 2y ago
My experiences have all been like this too. I am puzzled by how some people say it works for them
- simonw 2y ago
  I wrote this article precisely for people who are having trouble getting good results out of LLMs for coding: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
  - codegangsta 2y ago
    Just wanted to chime in and say how appreciative I’ve been about all your replies here, and overall content on AI. Your takes are super reasonable and well thought out.
  - 2y ago

kerkeslager 2y ago

It's worse than that. I see people say, "Look how great this is," and show me an example, and the example they show me is just not great. We're literally looking at the same thing, and they're excited that this LLM can do a college grads's job to the level of a third grader, and I'm just not excited about that.

kristopolous 2y ago

They're sophisticated tools at much as any other software. About 2 weeks ago I started on a streaming markdown parser for the terminal because none really existed. I've switched to human coding now but the first version was basically all llm prompting and a bunch of the code is still llm generated (maybe 80%). It's a parser, those are hard. There's stacks, states, lookaheads, look behinds, feature flags, color spaces, support for things like links and syntax highlighting... all forward streaming. Not easy https://github.com/kristopolous/Streamdown https://github.com/kristopolous/Streamdown

thisissomething 2y ago

What changed my point of view regarding LLMs was when I realized how crucial context is in increasing output quality. Treat the AI as a freelancer working on your project. How would you ask a freelancer to create a Kanban system for you? By simply asking "Create a Kanban system", or by providing them a 2-3 pages document describing features, guidelines, restrictions, requirements, dependencies, design ethos, etc? Which approach will get you closer to your objective? The same applies to LLM (when it comes to code generation). When well instructed, it can quickly generate a lot of working code, and apply the necessary fixes/changes you request inside that same context window. It still can't generate senior-level code, but it saves hours when doing grunt work or prototyping ideas. "Oh, but the code isn't perfect". Nor is the code of the average jr dev, but their codes still make it to production in thousands of companies around the world.

joquarky 2y ago
I see it as a knowledge multiplier. You still need to know enough about the subject to verify the output.