5 ms·

I very much disagree. To attempt a proof by contradiction: Let us assume that the author's premise is correct, and LLMs are plenty powerful given the right con

I very much disagree. To attempt a proof by contradiction: Let us assume that the author's premise is correct, and LLMs are plenty powerful given the right context. Can an LLM recognize the context deficit and frame the right questions to ask? They can not: LLMs have no ability to understand when to stop and ask for directions. They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc. They can not even reliably execute my "ok modify this text in canvas" vs "leave canvas alone, provide suggestions in chat, apply an edit once approved" instructions.

beering 1y ago
It feels crazy to keep arguing about LLMs being able to do this or that, but not mention the specific model? The post author only mentions the IMO gold-medal model. And your post could be about anything. Am I to believe that the two of you are talking about the same thing? This discussion is not useful if that’s not the case.
bobbylarrybobby 1y ago
Claude routinely stops and asks me clarifying questions before continuing, especially when the given extended thinking or doing research.
- diggan 1y ago
  Indeed, the ability to do so seems to depend more on how well your system prompt is laying out that workflow, than how "intelligent" the model is.
  - Jensson 1y ago
    Prompting it to ask clarifying questions will make it ask questions it has seen before, not ask questions it needs you to clarify. So that doesn't solve the problem, it just causes other problems. If it actually did solve the problem then they would train the models to act that way by default, so anything that you need to make smart prompts for has to be dumb.
  - JumpCrisscross 1y ago
    Haven’t we all met a smart person who never learned to think critically or in structured ways?
    - tharant 1y ago
      Hi, it’s me.
themanmaran 1y ago
This depends on whether you mean LLMs in the sense of single shot, or LLMs + software built around it. I think a lot of people conflate the two. In our application e use a multi-step check_knowledge_base workflow before and after each LLM request. Pretty much, make a separate LLM request to check the query against the existing context to see if more info is needed, and a second check after generation to see if output text exceeded it's knowledge base. And the results are really good. Now coding agents in your example are definitely stepwise more complex, but the same guardrails can apply.
- patrickhogan1 1y ago
  Do you have a concrete example of what you mean? For example, the article above was insightful. But the authors pointing to 1,000s of disparate workflows that could be solved with the right context, without actually providing 1 concrete example of how he accomplishes this makes the post weaker.
  - themanmaran 1y ago
    Sure, concrete example. We do conversational AI for banks, and spend a lot of time on the compliance side. Biggest thing is we don't want the LLM to ever give back an answer that could violate something like ECOA. So every message that gets generated by the first LLM is then passed to a second series of LLM requests + a distilled version of the legislation. ex: "Does this message imply likelihood of credit approval (True/False)". Then we can score the original LLM response based on that rubric. All of the compliance checks are very standardized, and have very little reasoning requirements, since they can mostly be distilled into a series of ~20 booleans.
    - patrickhogan1 1y ago
      Thank you! Great example!
- visarga 1y ago
  > Pretty much, make a separate LLM request to check the query against the existing context to see if more info is needed, and a second check after generation to see if output text exceeded it's knowledge base. They are unreliable at that. They can't reliably judge LLM outputs without access to the environment where those actions are executed and sufficient time to actually get to the outcomes that provide feedback signal. For example I was working on evaluation for an AI agent. The agent was about 80% correct, and the LLM judge about 80% accurate in assessing the agent. How can we have self correcting AI when it can't reliably self correct? Hence my idea - only the environment outcomes over a sufficient time span can validate work. But that is also expensive and risky.
bubblyworld 1y ago
This is not a proof by contradiction - you have stated an assumption followed by a bunch of non-sequitors about what LLMs can and can't do, also known as begging the question. Under the conditions of your assumption (namely that LLMs are plenty powerful with the right context) why would you believe anything in your last paragraph? That's how a proof by contradiction works. (not saying you are wrong, necessarily, but I don't think this argument holds water)
- therein 1y ago
  I agree it isn't really proof by contradiction. It is more like proof by demonstration of concrete failures in real life demonstrations, which is stronger. It is like the author is saying 12 is a prime number and I am like but I divided it by 2 just the other day.
  - bubblyworld 1y ago
    True, but in this case these are hardly globally applicable facts about LLM-based systems (not nearly to the same degree as "12 divides 2" anyway). Different systems have different properties on all those fronts. I don't think no argument is the right substitute for a bad one!
  - OrderlyTiamat 1y ago
    Nit pick, but proof by contradiction is necessarily stronger as it is deductive reasoning, and this kind of "proof" by anecdotal evidence doesn't rise above abductive reasoning. Still useful, very much not a proof.
    - woooooo 1y ago
      We don't have a formal model of how/why any given LLM works, and incidentally we're also short on proofs for real-world software and organizations. Empirical facts are the strongest thing we have in this domain.
      - giantg2 1y ago
TacticalCoder 1y ago
[dead]
Wowfunhappy 1y ago
> LLMs have no ability to understand when to stop and ask for directions. I haven't read TFA so I may be missing the point. However, I have had success getting Claude to stop and ask for directions by specifically prompting it to do so. "If you're stuck or the task seems impossible, please stop and explain the problem to me so I can help you."
- mrlongroots 1y ago
  Ok I think the confusion arises because of the probabilistic nature of LLM responses that blurs the line between "intelligent vs not". Let's take driving a car as an example, and a random decision generator as a lower bound on the intelligence of the driver. - A professionally trained human, who is not fatigued or unhealthy or substance-impaired, rarely makes a mistake, and when they do, there are reasonable mitigating factors. - ML models, OTOH, are very brittle and probabilistic. A model trained on blue tinted windshields may suffer a dramatic drop in performance if ran on yellow-tinted windshields. Models are unpredictably probabilistic. They do not learn a complete world model, but the very specific conditions and circumstances of their training dataset. They continue to get better, and you are able to induce a behavior similar to true intelligence more and more often. In your case, you are able to get them to stop and ask, but if they had the ability to do this reliably, they would not make mistakes as agents at all. Right now they resemble intelligence under a very specific light, and as the regimes under which they resemble one get bigger, they will get to AGIs. But we're not there yet.
antithesizer 1y ago
The word you're looking for is "rebuttal" since this is neither proof nor refutation of anything, but merely an argument against the thesis.
- mrlongroots 1y ago
  A rebuttal is just an alias for "counterargument", it does not define the structure of the counterargument. However flawed, what I said did have a structure (please refer to my other response in this thread for why).
ramesh31 1y ago
>They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc It's all about tools. Given sufficient tooling, the model's inherent abilities become irrelevant. Give a model a tool that counts characters and it will get this question right 100% of the time. Copy and paste to your domain. And what are tools but a means of providing context from the real world? People seem blinded by focusing on the raw abilities of models, missing the fact that these things should be seen simply as reasoning engines for tool usage.

beering 1y ago

It feels crazy to keep arguing about LLMs being able to do this or that, but not mention the specific model? The post author only mentions the IMO gold-medal model. And your post could be about anything. Am I to believe that the two of you are talking about the same thing? This discussion is not useful if that’s not the case.

bobbylarrybobby 1y ago

Claude routinely stops and asks me clarifying questions before continuing, especially when the given extended thinking or doing research.

diggan 1y ago
Indeed, the ability to do so seems to depend more on how well your system prompt is laying out that workflow, than how "intelligent" the model is.
- Jensson 1y ago
  Prompting it to ask clarifying questions will make it ask questions it has seen before, not ask questions it needs you to clarify. So that doesn't solve the problem, it just causes other problems. If it actually did solve the problem then they would train the models to act that way by default, so anything that you need to make smart prompts for has to be dumb.
- JumpCrisscross 1y ago
  Haven’t we all met a smart person who never learned to think critically or in structured ways?
  - tharant 1y ago
    Hi, it’s me.

themanmaran 1y ago

This depends on whether you mean LLMs in the sense of single shot, or LLMs + software built around it. I think a lot of people conflate the two. In our application e use a multi-step check_knowledge_base workflow before and after each LLM request. Pretty much, make a separate LLM request to check the query against the existing context to see if more info is needed, and a second check after generation to see if output text exceeded it's knowledge base. And the results are really good. Now coding agents in your example are definitely stepwise more complex, but the same guardrails can apply.

patrickhogan1 1y ago
Do you have a concrete example of what you mean? For example, the article above was insightful. But the authors pointing to 1,000s of disparate workflows that could be solved with the right context, without actually providing 1 concrete example of how he accomplishes this makes the post weaker.
- themanmaran 1y ago
  Sure, concrete example. We do conversational AI for banks, and spend a lot of time on the compliance side. Biggest thing is we don't want the LLM to ever give back an answer that could violate something like ECOA. So every message that gets generated by the first LLM is then passed to a second series of LLM requests + a distilled version of the legislation. ex: "Does this message imply likelihood of credit approval (True/False)". Then we can score the original LLM response based on that rubric. All of the compliance checks are very standardized, and have very little reasoning requirements, since they can mostly be distilled into a series of ~20 booleans.
  - patrickhogan1 1y ago
    Thank you! Great example!
visarga 1y ago
> Pretty much, make a separate LLM request to check the query against the existing context to see if more info is needed, and a second check after generation to see if output text exceeded it's knowledge base. They are unreliable at that. They can't reliably judge LLM outputs without access to the environment where those actions are executed and sufficient time to actually get to the outcomes that provide feedback signal. For example I was working on evaluation for an AI agent. The agent was about 80% correct, and the LLM judge about 80% accurate in assessing the agent. How can we have self correcting AI when it can't reliably self correct? Hence my idea - only the environment outcomes over a sufficient time span can validate work. But that is also expensive and risky.

bubblyworld 1y ago

This is not a proof by contradiction - you have stated an assumption followed by a bunch of non-sequitors about what LLMs can and can't do, also known as begging the question. Under the conditions of your assumption (namely that LLMs are plenty powerful with the right context) why would you believe anything in your last paragraph? That's how a proof by contradiction works. (not saying you are wrong, necessarily, but I don't think this argument holds water)

therein 1y ago
I agree it isn't really proof by contradiction. It is more like proof by demonstration of concrete failures in real life demonstrations, which is stronger. It is like the author is saying 12 is a prime number and I am like but I divided it by 2 just the other day.
- bubblyworld 1y ago
  True, but in this case these are hardly globally applicable facts about LLM-based systems (not nearly to the same degree as "12 divides 2" anyway). Different systems have different properties on all those fronts. I don't think no argument is the right substitute for a bad one!
- OrderlyTiamat 1y ago
  Nit pick, but proof by contradiction is necessarily stronger as it is deductive reasoning, and this kind of "proof" by anecdotal evidence doesn't rise above abductive reasoning. Still useful, very much not a proof.
  - woooooo 1y ago
    We don't have a formal model of how/why any given LLM works, and incidentally we're also short on proofs for real-world software and organizations. Empirical facts are the strongest thing we have in this domain.
    - giantg2 1y ago

TacticalCoder 1y ago

[dead]

Wowfunhappy 1y ago

> LLMs have no ability to understand when to stop and ask for directions. I haven't read TFA so I may be missing the point. However, I have had success getting Claude to stop and ask for directions by specifically prompting it to do so. "If you're stuck or the task seems impossible, please stop and explain the problem to me so I can help you."

mrlongroots 1y ago
Ok I think the confusion arises because of the probabilistic nature of LLM responses that blurs the line between "intelligent vs not". Let's take driving a car as an example, and a random decision generator as a lower bound on the intelligence of the driver. - A professionally trained human, who is not fatigued or unhealthy or substance-impaired, rarely makes a mistake, and when they do, there are reasonable mitigating factors. - ML models, OTOH, are very brittle and probabilistic. A model trained on blue tinted windshields may suffer a dramatic drop in performance if ran on yellow-tinted windshields. Models are unpredictably probabilistic. They do not learn a complete world model, but the very specific conditions and circumstances of their training dataset. They continue to get better, and you are able to induce a behavior similar to true intelligence more and more often. In your case, you are able to get them to stop and ask, but if they had the ability to do this reliably, they would not make mistakes as agents at all. Right now they resemble intelligence under a very specific light, and as the regimes under which they resemble one get bigger, they will get to AGIs. But we're not there yet.

antithesizer 1y ago

The word you're looking for is "rebuttal" since this is neither proof nor refutation of anything, but merely an argument against the thesis.

mrlongroots 1y ago
A rebuttal is just an alias for "counterargument", it does not define the structure of the counterargument. However flawed, what I said did have a structure (please refer to my other response in this thread for why).

ramesh31 1y ago

>They routinely produce contradictions, fail simple tasks like counting the letters in a word etc. etc It's all about tools. Given sufficient tooling, the model's inherent abilities become irrelevant. Give a model a tool that counts characters and it will get this question right 100% of the time. Copy and paste to your domain. And what are tools but a means of providing context from the real world? People seem blinded by focusing on the raw abilities of models, missing the fact that these things should be seen simply as reasoning engines for tool usage.