3 ms·
I think this is an epiphany everyone has to get through before making LLMs really useful. > An LLM wants to agree with both, it created plausible arguments for
by Aurornis 10mo ago
I think this is an epiphany everyone has to get through before making LLMs really useful.
> An LLM wants to agree with both, it created plausible arguments for both. While giving "caveats" instead of counterarguments.
My hypothesis is that LLMs are trained to be agreeable and helpful because many of their use cases involving taking orders and doing what the user wants. Additionally, some people and cultures have conversational styles where requests are phrased similarly to neutral questions to be polite.
It would be frustrating for users if they asked questions like “What do you think about having the background be blue?” and the LLM went off and said “Actually red is a more powerful color so I’m going to change it to red”. So my hypothesis is that the LLM training sets and training are designed to maximize agreeableness and having the LLM reflect tones and themes in the prompt, while discouraging disagreement. This is helpful when trying to get the LLM to do what you ask, but frustrating for anyone expecting a debate partner.
You can, however, build a pre-prompt that sets expectations for the LLM. You could even make a prompt asking it to debate everything with you, then to ask your questions.
- softwaredoug 10mo agoThe main "feature" of LLM is it has world-knowledge and can create plausible arguments for just about everything. "Nothing is true, Anything is possible" Which is a fascinating thing to think about epistemologically. Internally consistent knowledge of the LLM somehow can be used to create an argument for nearly anything. We humans think our cultural norms and truths are very special, that they're "obvious". But an LLM can create a fully formed counterfactual universe that sounds? is? just as plausible.
- ludicrousdispla 10mo agoAn LLM doesn't have any knowledge, it's just a masterful mix-tape machine.
- dgoodell 10mo agoWhat makes you think an LLM is internally consistent?
- Aurornis 10mo ago> But an LLM can create a fully formed counterfactual universe This is a little too far into the woo side of LLM interpretations. The LLM isn’t forming a universe internally. It’s stringing tokens together in a way that is consistent with language and something that looks coherent. It doesn’t hold opinions or have ideas about the universe that it has created from some first principles. It’s just a big statistical probability machine that was trained on the inputs we gave it.
- startupsfail 10mo agoThere are still blatant failure modes, when models engage into clear sycophancy, rather than expressing enthusiasm, etc. I'd guess, in practice a benchmark (like this vibesbench), that could help catching unhelpful and blatant sycophancy fails may help.
- satvikpendem 10mo ago> You could even make a prompt asking it to debate everything with you, then to ask your questions. This is exactly what I do, due to this sycophancy problem, and it works a lot better because it does not become agreeable with you but actively pushes back (sometimes so much so that I start getting annoyed with it, lol).