3 ms·

> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the p

> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the problem and why a paper like this is critical. These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc. People are being told it can do far more than it really is.

mjburgess 2y ago
Sure, but even these people... the failures are so common, and often very obvious. Consider a CEO who puts a press briefing in and asks some questions about it, it's not uncommon for those answers to be obviously wrong on any sort of critical reflection. We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. I guess that offers an explanation: people aren't aware that via their own prompt engineering they are repairing the deficiencies of the process by manipulating its inputs to include the structured reasoning it lacks. So there's a sort of hot-reading effect at work.
- throw46365 2y ago
  > We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. Right -- we are a long way from "this is a very nuanced error" being the dominant failure.
kalkin 2y ago
> People are being told it can do far more than it really is. Meanwhile these HN comments are split between: * Lots of people confirming what the paper itself notes (but doesn't highlight), that the most advanced models actually can solve this problem at least a significant portion of the time. (A proportion which one can pretty easily project is only likely to increase with future models.) * Lots of people saying "this confirms LLMs can't do reasoning". Questions I'd ask you to consider: * Is "LLMs can't do reasoning" actually more accurate than the typical hype? * Is a "critical understanding of how [LLMs] work" that would predict they simply cannot solve this problem actually a good understanding?
- mjburgess 2y ago
  The critical understanding doesnt predict that LLMs cannot solve problems. It predicts how they will solve them. There is no information, a priori, what the LLM has been trained on. You have to prompt, then see the answer. Once the answer arrives, the critical understanding provides a route to repairing the answer when not accurate or useful. LLMs do not reason. They appear to reason by repeating the structure of reasoning in their training data. This is indistinguishable in many cases.
  - travis86 2y ago
    I am very curious how you would define 'reasoning'.

mjburgess 2y ago

Sure, but even these people... the failures are so common, and often very obvious. Consider a CEO who puts a press briefing in and asks some questions about it, it's not uncommon for those answers to be obviously wrong on any sort of critical reflection. We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. I guess that offers an explanation: people aren't aware that via their own prompt engineering they are repairing the deficiencies of the process by manipulating its inputs to include the structured reasoning it lacks. So there's a sort of hot-reading effect at work.

throw46365 2y ago
> We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. Right -- we are a long way from "this is a very nuanced error" being the dominant failure.

kalkin 2y ago

> People are being told it can do far more than it really is. Meanwhile these HN comments are split between: * Lots of people confirming what the paper itself notes (but doesn't highlight), that the most advanced models actually can solve this problem at least a significant portion of the time. (A proportion which one can pretty easily project is only likely to increase with future models.) * Lots of people saying "this confirms LLMs can't do reasoning". Questions I'd ask you to consider: * Is "LLMs can't do reasoning" actually more accurate than the typical hype? * Is a "critical understanding of how [LLMs] work" that would predict they simply cannot solve this problem actually a good understanding?

mjburgess 2y ago
The critical understanding doesnt predict that LLMs cannot solve problems. It predicts how they will solve them. There is no information, a priori, what the LLM has been trained on. You have to prompt, then see the answer. Once the answer arrives, the critical understanding provides a route to repairing the answer when not accurate or useful. LLMs do not reason. They appear to reason by repeating the structure of reasoning in their training data. This is indistinguishable in many cases.
- travis86 2y ago
  I am very curious how you would define 'reasoning'.