3 ms·

LLMs cannot tell fact from fiction. What's commonly called hallucinations stems from it not being able to reason, the way that humans appear to be able to do, n

by lucb1e 1y ago

LLMs cannot tell fact from fiction. What's commonly called hallucinations stems from it not being able to reason, the way that humans appear to be able to do, no matter that some models are called "reasoning" now. It's all the same principle: most likely token in a given position. Adding internal monologue appears to help because, by being forced to break it down (internally, or by spitballing towards the user when they prompted "think step by step"[1]), it creates better context and will thus have a higher probability that the predicted token is a correct one Being trained to be positive is surely why it inserts these specific "great question, you're so right!" remarks, but if you wasn't trained on that, it still couldn't tell you whether you're great or not > I'm pretty sure they want it kissing people's asses The American faux friendliness is not what causes the underlying problem here, so all else being equal, they might as well have it kiss your ass. It's what most English speakers expect from a "friendly assistant" after all [1] https://hn.algolia.com/?dateEnd=1703980800&dateRange=custom&dateStart=1640995200&query=think%20step&sort=byPopularity&type=story https://hn.algolia.com/?dateEnd=1703980800&dateRange=custom&...

svnt 1y ago
You’re absolutely wrong! This is not how reasoning models work. Chain-of-thought did not produce reasoning models.
- lucb1e 1y ago
  Then I can't explain why it's producing the results that it does. If you have more information to share, I'm happy to update my knowledge... Doing a web search on the topic just comes up with marketing materials. Even Wikipedia's "Reasoning language model" article is mostly a list of release dates and model names, with as only relevant-sounding remark as to how these models are different: "[LLMs] can be fine-tuned on a dataset of reasoning tasks paired with example solutions and step-by-step (reasoning) traces. The fine-tuned model can then produce its own reasoning traces for new problems." It sounds like just another dataset: more examples, more training, in particular on worked examples where this "think step by step" method is being demonstrated with known-good steps and values. I don't see how that fundamentally changes how it works; you're saying such models do not predict the most likely token for a given context anymore, that there is some fundamentally different reasoning process going on somewhere?
  - svnt 1y ago
    I'm saying adding "think step by step" does not get you close to actual reasoning, it just produces marginally self-consistent linguistic reasoning. Actual reasoning requires training on diverse data sources, as you noted, but also coached experimentation (supervised fine-tuning) not just adding "think step by step" instruction to a model trained on typical textual datasets. "Think step by step" came first and produced increased performance on a variety of tasks, but was overhyped in its approximation of reasoning.
- Dylan16807 1y ago
  How do they work then? Because I thought chain of thought made for reasoning. And the first google result for 'chain of thought versus reasoning models' says it does: https://medium.com/@mayadakhatib/the-era-of-reasoning-models-moving-beyond-chain-of-thought-prompting-fa24ce25c106 https://medium.com/@mayadakhatib/the-era-of-reasoning-models... Give me a better source.
  - svnt 1y ago
    Did you even read the article you posted? It supports my statement. CoT produces the linguistic scaffolding for reasoning, but doesn't actually provide much accuracy in doing so. e.g. https://developer.nvidia.com/blog/maximize-robotics-performance-by-post-training-nvidia-cosmos-reason/ https://developer.nvidia.com/blog/maximize-robotics-performa...
    - Dylan16807 1y ago
      The article I posted says that chain of thought is enough to make something a reasoning model. It's listed as one of the types. It says that reasoning models "now aim for" more, but that doesn't disqualify the older more basic techniques. I'm not sure what I'm supposed to learn from your link because there's so much vision stuff happening here and that's the focus of the article. Can you link me something about what you need to make a bare minimum reasoning LLM? Text only.

by lucb1e 1y ago

svnt 1y ago

You’re absolutely wrong! This is not how reasoning models work. Chain-of-thought did not produce reasoning models.

lucb1e 1y ago
Then I can't explain why it's producing the results that it does. If you have more information to share, I'm happy to update my knowledge... Doing a web search on the topic just comes up with marketing materials. Even Wikipedia's "Reasoning language model" article is mostly a list of release dates and model names, with as only relevant-sounding remark as to how these models are different: "[LLMs] can be fine-tuned on a dataset of reasoning tasks paired with example solutions and step-by-step (reasoning) traces. The fine-tuned model can then produce its own reasoning traces for new problems." It sounds like just another dataset: more examples, more training, in particular on worked examples where this "think step by step" method is being demonstrated with known-good steps and values. I don't see how that fundamentally changes how it works; you're saying such models do not predict the most likely token for a given context anymore, that there is some fundamentally different reasoning process going on somewhere?
- svnt 1y ago
  I'm saying adding "think step by step" does not get you close to actual reasoning, it just produces marginally self-consistent linguistic reasoning. Actual reasoning requires training on diverse data sources, as you noted, but also coached experimentation (supervised fine-tuning) not just adding "think step by step" instruction to a model trained on typical textual datasets. "Think step by step" came first and produced increased performance on a variety of tasks, but was overhyped in its approximation of reasoning.
Dylan16807 1y ago
How do they work then? Because I thought chain of thought made for reasoning. And the first google result for 'chain of thought versus reasoning models' says it does: https://medium.com/@mayadakhatib/the-era-of-reasoning-models-moving-beyond-chain-of-thought-prompting-fa24ce25c106 https://medium.com/@mayadakhatib/the-era-of-reasoning-models... Give me a better source.
- svnt 1y ago
  Did you even read the article you posted? It supports my statement. CoT produces the linguistic scaffolding for reasoning, but doesn't actually provide much accuracy in doing so. e.g. https://developer.nvidia.com/blog/maximize-robotics-performance-by-post-training-nvidia-cosmos-reason/ https://developer.nvidia.com/blog/maximize-robotics-performa...
  - Dylan16807 1y ago
    The article I posted says that chain of thought is enough to make something a reasoning model. It's listed as one of the types. It says that reasoning models "now aim for" more, but that doesn't disqualify the older more basic techniques. I'm not sure what I'm supposed to learn from your link because there's so much vision stuff happening here and that's the focus of the article. Can you link me something about what you need to make a bare minimum reasoning LLM? Text only.