4 ms·

It appears that you are the only person in this discussion making many incorrect assumptions. Based on your comments, I would assume you are actually googling t

It appears that you are the only person in this discussion making many incorrect assumptions. Based on your comments, I would assume you are actually googling those papers based on their abstracts. Your last linked paper has flawed methodology for what it attempts to demonstrate, as shown in this paper: https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 The tests you're requesting are provided within the previously linked papers. I'm not sure what you want. Do you expect people to copy and paste entire papers here that show methodology and describe experiments? You wrote, "I'm asking you to define 'real reasoning'," which is actually defined in the blog post linked earlier in this discussion. In fact, the entire blog post is about this topic. It appears that you are not thoroughly reading the material. Your replies resemble those of a human stochastic parrot.

famouswaffles 2y ago
>Your last linked paper has flawed methodology for what it attempts to demonstrate, as shown in this paper: https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Genuinely, What's wrong with the methodology? Your paper literally admits humans would also perform worse at counterfactuals. Worse than a LLM ? Maybe not but it never bothers to test this so... The problem here is that none of the definitions (those that are testable) so far given actually separate humans from LLMs. They're all tests some humans would also flounder at or that LLMs perform far greater than chance at, if below some human's level. If you're going to say, "LLMs don't do real reasoning because of x" then x better be something all humans clear if what humans do is "real reasoning". Humans perform worse at counterfactuals so saying "Hey, see this paper that shows LLMs doing the same, It means they don't reason" is a logical fallacy if you don't extend that conclusion to humans as well.
- vidarh 2y ago
  In these arguments it's always very notable that not only do people not benchmark LLMs against people, but several I've discussed with have argued very strongly for not doing so unless they're benchmarked against above average people. While arguing that these same tests prove LLMs can reason. It never seems to land with them that their standards for "reason" would exclude large portions of the human population to some state of lesser being without the ability to reason.

famouswaffles 2y ago

>Your last linked paper has flawed methodology for what it attempts to demonstrate, as shown in this paper: https://arxiv.org/pdf/2307.02477 https://arxiv.org/pdf/2307.02477 Genuinely, What's wrong with the methodology? Your paper literally admits humans would also perform worse at counterfactuals. Worse than a LLM ? Maybe not but it never bothers to test this so... The problem here is that none of the definitions (those that are testable) so far given actually separate humans from LLMs. They're all tests some humans would also flounder at or that LLMs perform far greater than chance at, if below some human's level. If you're going to say, "LLMs don't do real reasoning because of x" then x better be something all humans clear if what humans do is "real reasoning". Humans perform worse at counterfactuals so saying "Hey, see this paper that shows LLMs doing the same, It means they don't reason" is a logical fallacy if you don't extend that conclusion to humans as well.

vidarh 2y ago
In these arguments it's always very notable that not only do people not benchmark LLMs against people, but several I've discussed with have argued very strongly for not doing so unless they're benchmarked against above average people. While arguing that these same tests prove LLMs can reason. It never seems to land with them that their standards for "reason" would exclude large portions of the human population to some state of lesser being without the ability to reason.