4 ms·

Many of my LLM experiences are similar in that they completely lie or make up functions in code or arguments to applications and only backtrack to apologize whe

Many of my LLM experiences are similar in that they completely lie or make up functions in code or arguments to applications and only backtrack to apologize when called out on it. Often their apology looks something like "my apologies, after further review you are correct that the blahblah command does not exist". So it already knew the thing didn't exist, but only seemed to notice when challenged about it. Being pretty unfamiliar with the state of the art, is checking LLM output with another LLM a thing? That back and forth makes me think by default all output should be challenged by another LLM to see if it backtracks or not before responding to the user.

michaelt 1y ago
As I understand things, part of what you get with these coding agents is automating the process of 1. LLM writes broken code, such as using an imaginary function, 2. user compiles/runs the code and it errors because the function doesn't exist, 3. paste the error message into the LLM, 4. LLM tries to fix the error, 5. Loop. Much like a company developing a new rocket by launching, having it explode, fixing the cause of that explosion, then launching another rocket, in a loop until their rockets eventually stop exploding. I don't connect my live production database to what I think of as an exploding rocket, and I find it bewildering that apparently other people do....
- ehnto 1y ago
  The trouble is that it won't actually learn from its mistakes, and often in business the mistakes are very particular to your processes such that they will never be in the training data. So when the agent attempts to codify the business logic you need to be super specific, and there are many businesses I have worked in where it is just too complex and arbitrary for an LLM to keep the thread reliably. Even when you feed it all the business requirements. Maybe this changes over time but as I work with it now, there is an intrinsic limitation to how nuanced they can be without getting confused.
ehnto 1y ago
> So it already knew the thing didn't exist, but only seemed to notice when challenged about it. This backfilling of information or logic is the most frustrating part of working with LLMs. When using agents I usually ask it to double check its work.
lionkor 1y ago
It didn't "know" anything. That's not even remotely how LLMs work.
- johnisgood 1y ago
  Nor does it ever "lie". To lie is to intentionally deceive.
  - recursive 1y ago
    Why do you say they cannot have intention?
    - johnisgood 1y ago
      Because intent implies: 1. goal-directedness, 2. mental states (beliefs, desires, motivations), and 3. consciousness or awareness. LLMs lack intent because 1) they have no goals of their own. They do not "want" anything, they do not form desires, 2) they have no mental states (they can simulate language about them, but do not actually posses them, and 3) they are not conscious. They do not experience, reflect, or understand in the way that conscious beings do. Thus, under the philosophical and cognitive definition, LLMs do not have intent. They can mimic intent, the same way a thermostat is "trying" to keep a room at a certain temperature, but it is only apparent or simulated intent, not genuine intent we ascribe to humans.
      - recursive 1y ago
        LLMs can make false statements. The distinction about real vs simulated intent doesn't seem useful. LLMs can have objectives. Those objectives can sometimes be advanced via deception. Many people call this kind of deception (and sometimes others) lying. If we have these words which apply to humans only, definitionally, then we're going to need some new words or some new definitions. We don't really have a way to talk about what's going on here. Personally, I'm fine using "lying".
        1y ago

michaelt 1y ago

As I understand things, part of what you get with these coding agents is automating the process of 1. LLM writes broken code, such as using an imaginary function, 2. user compiles/runs the code and it errors because the function doesn't exist, 3. paste the error message into the LLM, 4. LLM tries to fix the error, 5. Loop. Much like a company developing a new rocket by launching, having it explode, fixing the cause of that explosion, then launching another rocket, in a loop until their rockets eventually stop exploding. I don't connect my live production database to what I think of as an exploding rocket, and I find it bewildering that apparently other people do....

ehnto 1y ago
The trouble is that it won't actually learn from its mistakes, and often in business the mistakes are very particular to your processes such that they will never be in the training data. So when the agent attempts to codify the business logic you need to be super specific, and there are many businesses I have worked in where it is just too complex and arbitrary for an LLM to keep the thread reliably. Even when you feed it all the business requirements. Maybe this changes over time but as I work with it now, there is an intrinsic limitation to how nuanced they can be without getting confused.

ehnto 1y ago

> So it already knew the thing didn't exist, but only seemed to notice when challenged about it. This backfilling of information or logic is the most frustrating part of working with LLMs. When using agents I usually ask it to double check its work.

lionkor 1y ago

It didn't "know" anything. That's not even remotely how LLMs work.

johnisgood 1y ago
Nor does it ever "lie". To lie is to intentionally deceive.
- recursive 1y ago
  Why do you say they cannot have intention?
  - johnisgood 1y ago
    Because intent implies: 1. goal-directedness, 2. mental states (beliefs, desires, motivations), and 3. consciousness or awareness. LLMs lack intent because 1) they have no goals of their own. They do not "want" anything, they do not form desires, 2) they have no mental states (they can simulate language about them, but do not actually posses them, and 3) they are not conscious. They do not experience, reflect, or understand in the way that conscious beings do. Thus, under the philosophical and cognitive definition, LLMs do not have intent. They can mimic intent, the same way a thermostat is "trying" to keep a room at a certain temperature, but it is only apparent or simulated intent, not genuine intent we ascribe to humans.
    - recursive 1y ago
      LLMs can make false statements. The distinction about real vs simulated intent doesn't seem useful. LLMs can have objectives. Those objectives can sometimes be advanced via deception. Many people call this kind of deception (and sometimes others) lying. If we have these words which apply to humans only, definitionally, then we're going to need some new words or some new definitions. We don't really have a way to talk about what's going on here. Personally, I'm fine using "lying".
      - 1y ago