4 ms·
I'm not the person you responded to, but that's an old paper. LLMs have come a long way since. Nothing conclusive but GPT4 does show some signs of something dee
by mgfist 4y ago
I'm not the person you responded to, but that's an old paper. LLMs have come a long way since. Nothing conclusive but GPT4 does show some signs of something deeper. https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
- cscurmudgeon 4y agoYes, I have read that paper and have forwarded it eagerly to colleagues. Respectfully, please read the paper you have linked, especially section 8.2. Look at section 8.2 in the paper you linked: "8.2 Lack of planning in arithmetic/reasoning problems" > However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution Nowhere in the paper is an evaluation on standard reasoning datasets (https://tptp.org https://tptp.org) because they are much much much tougher than the simple examples that GPT-4 struggles with. LLMs are a huge breakthrough but let us not muddy the science.
- Sunhold 4y agoGPT 4 was evaluated on multiple reasoning benchmarks in the release paper.[1] The problems in TPTP are not really appropriate for a language model and exceed the capabilities of most humans working without tools. Clearly humans have some reasoning ability even if they cannot solve those problems. [1] https://openai.com/research/gpt-4 https://openai.com/research/gpt-4