2 ms·
and how can you tell they reason and not parrot some text in training data? There are papers about trying LLMs on generated reasoning problems, and they usuall
by riku_iki 3y ago
and how can you tell they reason and not parrot some text in training data?
There are papers about trying LLMs on generated reasoning problems, and they usually fail.
- nuancebydefault 3y ago>Usually That implies - sometimes not. Which would prove at least some reasoning capabilities.
- riku_iki 3y agoIn this case I used 'usually' because don't remember all details and didn't want to generalize by saying 'always', but also training/benchmarking protocol can be flawed, for example LLM still can solve shallow reasoning problem by memorizing pattern.
- gnaritas99 3y ago[dead]