4 ms·

You can just tell the agent to do exactly that

by wagwang 5mo ago

alterom 5mo ago
>You can just tell the agent to do exactly that You can. It just won't do it.
- wagwang 5mo ago
  Seems to work for me https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f658887 https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f6588...
tempest_ 5mo ago
I've had various agents backed by various models ignore the shit out of various rules and request at varying rates but they all do it. When you point it out "Oh yes, I did do that which is contrary to the rules, request <whatever>.. Anyway..."
- wagwang 5mo ago
  If you are on a sota model and your context window is less than 100k tokens and you don't have any vague or contradicting rules, then I've almost never seen a rule broken The most common failure I've seen come from tools that pollute their context with crap and the llm will forget stuff or just get confused from all the irrelevant sentences; which if the report is true, is probably what these ai notetakers are guilty of. This problem gets exacerbated if these tools turn on the 1M context window version.
  - fg137 5mo ago
    Yeah, that's exactly why I have full confidence in that system, especially for medical notetaking. /s
D-Machine 5mo ago
Except you can't be sure it isn't producing nonsense when you do this, and generally the model(s) will be overconfident. This has been studied, see e.g. https://openreview.net/pdf?id=E6LOh5vz5x https://openreview.net/pdf?id=E6LOh5vz5x > An alternative way to obtain uncertainty estimates from LLMs is to prompt them directly. One benefit of this approach is that it requires no access to the internals of the model. However, this approach has produced mixed results: LLMs can sometimes verbalize calibrated confidence levels (Lin et al., 2022a; Tian et al., 2023), but can also be highly overconfident (Xiong et al., 2024). Interestingly, Xiong et al. (2024) found that LLMs typically state confidence values in the range of 80-100%, usually in multiples of 5, potentially in imitation of how humans discuss confidence levels. Nevertheless, prompting strategies remain an important tool for uncertainty quantification, along with measures based on the internal state (such as MSP).

You can just tell the agent to do exactly that

by wagwang 5mo ago

You can just tell the agent to do exactly that

alterom 5mo ago

>You can just tell the agent to do exactly that You can. It just won't do it.

wagwang 5mo ago
Seems to work for me https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f658887 https://chatgpt.com/share/6a06a4c5-d454-83e8-a5b2-c9468f6588...

tempest_ 5mo ago

I've had various agents backed by various models ignore the shit out of various rules and request at varying rates but they all do it. When you point it out "Oh yes, I did do that which is contrary to the rules, request <whatever>.. Anyway..."

wagwang 5mo ago
If you are on a sota model and your context window is less than 100k tokens and you don't have any vague or contradicting rules, then I've almost never seen a rule broken The most common failure I've seen come from tools that pollute their context with crap and the llm will forget stuff or just get confused from all the irrelevant sentences; which if the report is true, is probably what these ai notetakers are guilty of. This problem gets exacerbated if these tools turn on the 1M context window version.
- fg137 5mo ago
  Yeah, that's exactly why I have full confidence in that system, especially for medical notetaking. /s

D-Machine 5mo ago

Except you can't be sure it isn't producing nonsense when you do this, and generally the model(s) will be overconfident. This has been studied, see e.g. https://openreview.net/pdf?id=E6LOh5vz5x https://openreview.net/pdf?id=E6LOh5vz5x > An alternative way to obtain uncertainty estimates from LLMs is to prompt them directly. One benefit of this approach is that it requires no access to the internals of the model. However, this approach has produced mixed results: LLMs can sometimes verbalize calibrated confidence levels (Lin et al., 2022a; Tian et al., 2023), but can also be highly overconfident (Xiong et al., 2024). Interestingly, Xiong et al. (2024) found that LLMs typically state confidence values in the range of 80-100%, usually in multiples of 5, potentially in imitation of how humans discuss confidence levels. Nevertheless, prompting strategies remain an important tool for uncertainty quantification, along with measures based on the internal state (such as MSP).