3 ms·

Just 2 days ago Gemini 2.5 Pro tried to recommend me tax evasion based on non-existing laws and court decisions. The model was so charming and convincing, that

by nicce 1y ago

Just 2 days ago Gemini 2.5 Pro tried to recommend me tax evasion based on non-existing laws and court decisions. The model was so charming and convincing, that even after I brought all the logic flaws and said that this is plain wrong, I started to doubt myself, because it is so good at pleasing, arguing and using words. And most would have accept the recommendation because the model sold it as less common tactic, while sounding very logical.

roywiggins 1y ago
> even after I brought all the logic flaws and said that this is plain wrong Once you've started to argue with an LLM you're already barking up the wrong tree. Maybe you're right, maybe not, but there's no point in arguing it out with an LLM.
- nicce 1y ago
  There are cases when they are actually correct, instead of the human.
  - roywiggins 1y ago
    Yes, and there's a substantial chance they'll apologize to you anyway even when they were right. There's no reason to expect them to be more likely to apologize when they're actually right vs actually wrong- their agreeableness is really orthogonal to their correctness.
    - nicce 1y ago
      Yes, they over-apologize. But my main reason for using LLMs is seeking out things that I missed myself or my own argumentation was not good. Sometimes they are really good at bringing new perspectives. Whether they are correct or incorrect is not the point - are they giving argument or perspective that is worth inspecting more with my own brains?
nutjob2 1y ago
Or you could understand the tool you are using and be skeptical of any of its output. So many people just want to believe, instead of the reality of LLMs being quite unreliable. Personally it's usually fairly obvious to me when LLMs are bullshitting probably because I have lots of experience detecting it in humans.
- nicce 1y ago
  LLM is only useful if it gives shortcut to information with reasonable accuracy. If I need to double check everything, it is just extra step. In this case I just happened to be domain expert and knew it was wrong. It would have required significant effort to verify everything with some less experienced person.

by nicce 1y ago

roywiggins 1y ago

> even after I brought all the logic flaws and said that this is plain wrong Once you've started to argue with an LLM you're already barking up the wrong tree. Maybe you're right, maybe not, but there's no point in arguing it out with an LLM.

nicce 1y ago
There are cases when they are actually correct, instead of the human.
- roywiggins 1y ago
  Yes, and there's a substantial chance they'll apologize to you anyway even when they were right. There's no reason to expect them to be more likely to apologize when they're actually right vs actually wrong- their agreeableness is really orthogonal to their correctness.
  - nicce 1y ago
    Yes, they over-apologize. But my main reason for using LLMs is seeking out things that I missed myself or my own argumentation was not good. Sometimes they are really good at bringing new perspectives. Whether they are correct or incorrect is not the point - are they giving argument or perspective that is worth inspecting more with my own brains?

nutjob2 1y ago

Or you could understand the tool you are using and be skeptical of any of its output. So many people just want to believe, instead of the reality of LLMs being quite unreliable. Personally it's usually fairly obvious to me when LLMs are bullshitting probably because I have lots of experience detecting it in humans.

nicce 1y ago
LLM is only useful if it gives shortcut to information with reasonable accuracy. If I need to double check everything, it is just extra step. In this case I just happened to be domain expert and knew it was wrong. It would have required significant effort to verify everything with some less experienced person.