3 ms·

I appreciate the value of AI in my own work, but "they’re very good coders" and "they make trivial mistakes or hallucinate" seems incongruous. For me, AI tools

I appreciate the value of AI in my own work, but "they’re very good coders" and "they make trivial mistakes or hallucinate" seems incongruous. For me, AI tools shine when I know enough about a topic to quickly error check, but not enough where I can code fluently without documentation. I'm sure it will get more useful over time, but that's where it's been for me for the last year or so.

saretup 1y ago
> "they’re very good coders" and "they make trivial mistakes or hallucinate" seems incongruous. It seems that way if you judge them the same way you would human coders, but they’re different. They might be able to do things that veteran coders can’t without spending days on it, and fail at things that beginners can do in half an hour. A car might not be able to traverse difficult terrain as well as a horse, but it doesn’t mean the car is not a good mode of transportation.
- a_bonobo 1y ago
  Yes, human coders are predictably 'bad' - it scales roughly with the complexity of the task and the years of their experience, along with some randomness ('brain farts'). LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field. The unpredictability is what makes these tools so unfit for purpose.
  - groestl 1y ago
    > The unpredictability is what makes these tools so unfit for purpose. Also, their eagerness to produce. A good employee (senior or not!), knows when they're at the end of their wits and say so. In areas where LLMs are knowledgeable, but not knowledgeable enough, they'll still produce. They're stuck at unconscious incompetence, kind of like a systematic Dunning-Kruger effect. To solve that, we can either increase competence (moving out the area where this effect is noticeable), or maybe introduce some kind of feedback, self reflection (one could say, some kind of conciousness [0]). [0] inspired by "Goedel, Escher, Bach"
  - JimDabell 1y ago
    > LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field. That doesn’t mean they are unpredictably bad, it means assuming that they behave like humans is a bad way of predicting their behaviour. “Unlike humans” and “unpredictable” are not the same thing. If you spend time working with them, you get a better sense of what they are good and bad at and get better at predicting their behaviour.
- sampullman 1y ago
  I'm not sure if the analogy fits. I would say AI is more like a fast car that turns the wrong way every couple of blocks, or a car that randomly refuses to go on certain roads. Is it still a useful mode of transportation? Sometimes, but, it's not reliable. The sibling comment says it better - it's about predicability of mistakes, and the effort and knowledge necessary to correct them. Maybe the next generation of models will solve these problems, who knows.
  - fsflover 1y ago
    > AI is more like a fast car that turns the wrong way every couple of blocks So if you drive well and can control this "car", you are able to fix that and benefit from the fast "car" a lot. > or a car that randomly refuses to go on certain roads This doesn't seem like a good comparison. AI will do anything, it may just be wrong.
    - sampullman 1y ago
      The AIs that I use will definitely not do anything, and it the limits are not always obvious without a lot of prodding. I'm fine with ditching the car analogy though.
- amelius 1y ago
  The problem I have with AI is that you cannot reason with them after they make mistakes. They always reply with "Good point!" and then they give me another iteration of bad fixes. You really have to know when you should give up and do things manually.
  - chairhairair 1y ago
    Gemini 2.5 is much more confident. Anecdotally, I think this behavior is undesirable for most commercial LLM use cases. I have several friends that have complained about Gemini’s “back talking” and prefer ChatGPT’s relative sycophancy.
    - amelius 1y ago
      I was talking about ChatGpt, but other llms also have this problem. I don't really care how polite they are, etc., I just want to get the job done, but the llm often gets stuck at some point.

saretup 1y ago

> "they’re very good coders" and "they make trivial mistakes or hallucinate" seems incongruous. It seems that way if you judge them the same way you would human coders, but they’re different. They might be able to do things that veteran coders can’t without spending days on it, and fail at things that beginners can do in half an hour. A car might not be able to traverse difficult terrain as well as a horse, but it doesn’t mean the car is not a good mode of transportation.

a_bonobo 1y ago
Yes, human coders are predictably 'bad' - it scales roughly with the complexity of the task and the years of their experience, along with some randomness ('brain farts'). LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field. The unpredictability is what makes these tools so unfit for purpose.
- groestl 1y ago
  > The unpredictability is what makes these tools so unfit for purpose. Also, their eagerness to produce. A good employee (senior or not!), knows when they're at the end of their wits and say so. In areas where LLMs are knowledgeable, but not knowledgeable enough, they'll still produce. They're stuck at unconscious incompetence, kind of like a systematic Dunning-Kruger effect. To solve that, we can either increase competence (moving out the area where this effect is noticeable), or maybe introduce some kind of feedback, self reflection (one could say, some kind of conciousness [0]). [0] inspired by "Goedel, Escher, Bach"
- JimDabell 1y ago
  > LLMs are completely unpredictably 'bad' - sometimes a task that a high schooler can solve sends an LLM into a spin, sometimes an LLM solves a task that would require a PhD in the field. That doesn’t mean they are unpredictably bad, it means assuming that they behave like humans is a bad way of predicting their behaviour. “Unlike humans” and “unpredictable” are not the same thing. If you spend time working with them, you get a better sense of what they are good and bad at and get better at predicting their behaviour.
sampullman 1y ago
I'm not sure if the analogy fits. I would say AI is more like a fast car that turns the wrong way every couple of blocks, or a car that randomly refuses to go on certain roads. Is it still a useful mode of transportation? Sometimes, but, it's not reliable. The sibling comment says it better - it's about predicability of mistakes, and the effort and knowledge necessary to correct them. Maybe the next generation of models will solve these problems, who knows.
- fsflover 1y ago
  > AI is more like a fast car that turns the wrong way every couple of blocks So if you drive well and can control this "car", you are able to fix that and benefit from the fast "car" a lot. > or a car that randomly refuses to go on certain roads This doesn't seem like a good comparison. AI will do anything, it may just be wrong.
  - sampullman 1y ago
    The AIs that I use will definitely not do anything, and it the limits are not always obvious without a lot of prodding. I'm fine with ditching the car analogy though.
amelius 1y ago
The problem I have with AI is that you cannot reason with them after they make mistakes. They always reply with "Good point!" and then they give me another iteration of bad fixes. You really have to know when you should give up and do things manually.
- chairhairair 1y ago
  Gemini 2.5 is much more confident. Anecdotally, I think this behavior is undesirable for most commercial LLM use cases. I have several friends that have complained about Gemini’s “back talking” and prefer ChatGPT’s relative sycophancy.
  - amelius 1y ago
    I was talking about ChatGpt, but other llms also have this problem. I don't really care how polite they are, etc., I just want to get the job done, but the llm often gets stuck at some point.