3 ms·

I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain D

I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain Dew instead of Sprite if you have a southern accent sounds like a nightmare.

geysersam 2y ago
I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
- otabdeveloper4 2y ago
  > and then ask it "why did you just now give that customer mountain dew when they ordered sprite?" Worse than useless for debugging. An LLM can't think and doesn't have capabilities for self-reflection. It will just generate a plausible stream of tokens in reply that may or may not correspond to the real reason why.
  - geysersam 2y ago
    Of course a llm can't think. But that doesn't mean it can't answer simple questions about the output that was produced. Just try it out with chatgpt when you have time. Even if it's not perfectly accurate it's still useful for debugging. Just think about it as a human employee. Can they always say why they did what they did? Often, but not always. Sometimes you will have to work to figure out the misunderstanding.
    - otabdeveloper4 2y ago
      > it's still useful for debugging How so? What the LLM says is whatever is more likely given the context. It has no relation to the underlying reality whatsoever.
      - geysersam 2y ago
        Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I answered that your bike is at home because the last action you described was biking home, which implies you took the bike with you and ended your journey there. Therefore, the bike would logically be at home now. Do you doubt that the answer would change if I changed the query to make the final destination be "the park" instead of "home"? If you don't doubt that, what do you mean that the answer doesn't correspond to the underlying reality? The reality is the answer depends on the final destination mentioned, and that's also the explanation given by the LLM, clearly the reality and the answers are related.
        janalsncm 2y ago
        You need to find an example of the LLM making a mistake. In your example, ChatGPT answered correctly. There are many examples online of LLMs answering basic questions incorrectly, and then the person asking the LLM why it did so. The LLM response is usually nonsense. Then there is the question of what you would do with its response. It’s not like code where you can go in and update the logic. There are billions of floating point numbers. If you actually wanted to update the weights you’ll quickly find yourself fine-tuning the monstrosity. Orders of magnitude more work than updating an “if” statement.
        geysersam 2y ago
        I don't think llms always can give correct explanations for their answers. That's a misunderstanding. > Then there is the question of what you would do with its response. I Sure but that's a separate question. I'd say the first course of action would be to edit the prompt. If you have to resort to fine tuning I'd say the approach has failed and the tool was insufficient for the task.
- janalsncm 2y ago
  Why not just store the state in the code and debug as usual, perhaps with LLM assistance? At least that’s tractable.

geysersam 2y ago

I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"

otabdeveloper4 2y ago
> and then ask it "why did you just now give that customer mountain dew when they ordered sprite?" Worse than useless for debugging. An LLM can't think and doesn't have capabilities for self-reflection. It will just generate a plausible stream of tokens in reply that may or may not correspond to the real reason why.
- geysersam 2y ago
  Of course a llm can't think. But that doesn't mean it can't answer simple questions about the output that was produced. Just try it out with chatgpt when you have time. Even if it's not perfectly accurate it's still useful for debugging. Just think about it as a human employee. Can they always say why they did what they did? Often, but not always. Sometimes you will have to work to figure out the misunderstanding.
  - otabdeveloper4 2y ago
    > it's still useful for debugging How so? What the LLM says is whatever is more likely given the context. It has no relation to the underlying reality whatsoever.
    - geysersam 2y ago
      Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I answered that your bike is at home because the last action you described was biking home, which implies you took the bike with you and ended your journey there. Therefore, the bike would logically be at home now. Do you doubt that the answer would change if I changed the query to make the final destination be "the park" instead of "home"? If you don't doubt that, what do you mean that the answer doesn't correspond to the underlying reality? The reality is the answer depends on the final destination mentioned, and that's also the explanation given by the LLM, clearly the reality and the answers are related.
      - janalsncm 2y ago
        You need to find an example of the LLM making a mistake. In your example, ChatGPT answered correctly. There are many examples online of LLMs answering basic questions incorrectly, and then the person asking the LLM why it did so. The LLM response is usually nonsense. Then there is the question of what you would do with its response. It’s not like code where you can go in and update the logic. There are billions of floating point numbers. If you actually wanted to update the weights you’ll quickly find yourself fine-tuning the monstrosity. Orders of magnitude more work than updating an “if” statement.
        geysersam 2y ago
        I don't think llms always can give correct explanations for their answers. That's a misunderstanding. > Then there is the question of what you would do with its response. I Sure but that's a separate question. I'd say the first course of action would be to edit the prompt. If you have to resort to fine tuning I'd say the approach has failed and the tool was insufficient for the task.
janalsncm 2y ago
Why not just store the state in the code and debug as usual, perhaps with LLM assistance? At least that’s tractable.