3 ms·

I think there's two different cases here that need to be treated carefully when working with AI: 1. Using a well know but complex algorithm that I don't rememb

I think there's two different cases here that need to be treated carefully when working with AI: 1. Using a well know but complex algorithm that I don't remember fully. AI will know it and integrate it into my existing code faster (often much, much faster) than I could, and then I can review and confirm it's correct 2. Developing a new algorithm or at least novel application of an existing one, or using a complex algorithm in an unusual way. The AI will need a lot of guidance here, and often I'll regret asking it in the first place. I haven't used Claude Code, however every time I've criticized AI in the past, there's always someone who will say "this tool released in the last month totally fixes everything!"... And so far they haven't been correct. But the tools are getting better, so maybe this time it's true. $200 a month is a big ask though, completely out of reach for most people on earth (students, hobbyists, people from developing countries where it's close to a monthly wage) so I hope it doesn't become normalized.

somenameforme 1y ago
> I haven't used Claude Code, however every time I've criticized AI in the past, there's always someone who will say "this tool released in the last month totally fixes everything!"... And so far they haven't been correct. But the tools are getting better, so maybe this time it's true. The cascading error problem means this will probably never be true. Because LLMs are fundamentally guess the next token based on the previous tokens, whenever it gets a single token wrong - future tokens become even more likely to be wrong which snowballs to absurdity. Extreme hallucination issues can probably eventually be resolved by giving it access to a compiler and, where appropriate, you could also probably feed it test cases, but I don't think the cascading errors will ever be able to be resolved. The best case scenario will eventually it being able to say 'I don't know how to achieve this.' Of course then you ruin the mystique of LLMs which think they can solve any problem.
- HeatrayEnjoyer 1y ago
  It obviously can be resolved, otherwise we wouldn't be able to self-correct our own selves. When is unknown, but not the if.
  - raddan 1y ago
    We can sometimes correct ourselves. With training, in specific circumstances. The same insight (given enough time, a coding agent will make a mistake) is true for even the best human programmers, and I don’t see any mechanism that would make an LLM different.
    - somenameforme 1y ago
      The reason you will basically never just recommend e.g. somebody use a completely nonexistent function is because you're not just guessing what the answer to something should be. Rather you have a knowledge base which you believe to be correct and are constantly evolving and drawing from it. LLMs do not function like this at all. Rather all they have is a series of weights to help predict the next token given the prior tokens. Cascading errors is a lot like a math problem. If you make a mistake somewhere along when solving a lengthy problem then your further calculations will also continue to be more and more wrong. The same is true of an LLM when executing its prediction algorithm. This is why an LLM does give you a wrong answer it's usually just an exercise in frustration trying to get it to correct itself, and you'd be better of just creating a completely new context.
  - somenameforme 1y ago
    We aren't LLMs, obviously.

somenameforme 1y ago

> I haven't used Claude Code, however every time I've criticized AI in the past, there's always someone who will say "this tool released in the last month totally fixes everything!"... And so far they haven't been correct. But the tools are getting better, so maybe this time it's true. The cascading error problem means this will probably never be true. Because LLMs are fundamentally guess the next token based on the previous tokens, whenever it gets a single token wrong - future tokens become even more likely to be wrong which snowballs to absurdity. Extreme hallucination issues can probably eventually be resolved by giving it access to a compiler and, where appropriate, you could also probably feed it test cases, but I don't think the cascading errors will ever be able to be resolved. The best case scenario will eventually it being able to say 'I don't know how to achieve this.' Of course then you ruin the mystique of LLMs which think they can solve any problem.

HeatrayEnjoyer 1y ago
It obviously can be resolved, otherwise we wouldn't be able to self-correct our own selves. When is unknown, but not the if.
- raddan 1y ago
  We can sometimes correct ourselves. With training, in specific circumstances. The same insight (given enough time, a coding agent will make a mistake) is true for even the best human programmers, and I don’t see any mechanism that would make an LLM different.
  - somenameforme 1y ago
    The reason you will basically never just recommend e.g. somebody use a completely nonexistent function is because you're not just guessing what the answer to something should be. Rather you have a knowledge base which you believe to be correct and are constantly evolving and drawing from it. LLMs do not function like this at all. Rather all they have is a series of weights to help predict the next token given the prior tokens. Cascading errors is a lot like a math problem. If you make a mistake somewhere along when solving a lengthy problem then your further calculations will also continue to be more and more wrong. The same is true of an LLM when executing its prediction algorithm. This is why an LLM does give you a wrong answer it's usually just an exercise in frustration trying to get it to correct itself, and you'd be better of just creating a completely new context.
- somenameforme 1y ago
  We aren't LLMs, obviously.