5 ms·

> their function is to produce text output which forms a plausible seeming response to the question posed Answering "I don't know" or "I can't answer that" is

> their function is to produce text output which forms a plausible seeming response to the question posed Answering "I don't know" or "I can't answer that" is a perfectly plausible response to a difficult logical problem/question. And it would not be a hallucination.

dragonwriter 3y ago
> Answering "I don't know" or "I can't answer that" is a perfectly plausible response to a difficult logical problem/question. Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I couldn't solve”. And that’s why LLMs behave the way they do.
- someplaceguy 3y ago
  > Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I couldn't solve”. Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? Sure, there will be an error rate, but surely you can train an LLM to minimize it?
  - dragonwriter 3y ago
    > Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? No, I am saying that they are specific trained to do that, and that the results seen in practice on common real-world LLMs reflect the bias of the specific training they are given for providing concrete answers. > Sure, there will be an error rate, but surely you can train an LLM to minimize it? Giving some answer to a question that cannot be infallibly solved analytically is not necessarily an error. In fact, I would argue that providing useful answers in situations like that is among the motivating use cases for AI. (Whether or not the answers current LLMs provide in these cases are useful is another question, but you miss 100% of the shots you don’t take.)
  - pixl97 3y ago
    >Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? This is highly problematic and highly contextualized statement. Imagine you're an accountant with the piece of information $x. The answer you give for the statement "What is $x" is going to be highly dependent on who is answering the question. For example 1. The CEO asks "What is $x" 2. A regulator at the SEC asks "What is $x 3. Some random individual or member of the press asks "What is $x" An LLM doesn't have the other human motivations a person does when asked questions, pretty much at this point with LLMs there are only one or two 'voices' it hears (system prompt and user messages). Whereas a human will commonly lie and say I don't know, it's somewhat questionable if we want LLMs intentionally lying. In addition human information is quite often compartmentalized to keep secrets which is currently not in vogue with LLMs as we are attempting to make oracles that know everything with them.
    - someplaceguy 3y ago
      > The answer you give for the statement "What is $x" is going to be highly dependent on who is answering the question. I assume you meant asking rather than answering? > An LLM doesn't have the other human motivations a person does when asked questions, pretty much at this point with LLMs there are only one or two 'voices' it hears (system prompt and user messages). Why would LLMs need any motivation besides how they are trained to be helpful and the given prompts? In my experience with ChatGPT 4, it seems to be pretty good at discerning what and how to answer based on the prompts and context alone. > Whereas a human will commonly lie and say I don't know, it's somewhat questionable if we want LLMs intentionally lying. Why did you jump to the conclusion that an LLM answering "I don't know" is lying? I want LLMs to answer "I don't know" when they don't have enough information to provide a true answer. That's not lying, in fact it's the opposite, because the alternative is to hallucinate an answer. Hallucinations are the "lies" in this scenario. > In addition human information is quite often compartmentalized to keep secrets which is currently not in vogue with LLMs as we are attempting to make oracles that know everything with them. I'd rather have an oracle that can discriminate when it doesn't have enough information to provide a true answer and replies "I don't know" in such cases (or sometimes answer like "If I were to guess, then bla bla bla, but I'm not sure about this"), than one which always gives confident but sometimes wrong answers.
- paulmd 3y ago
  if more guiderails are useful to users then such things will surely emerge. but from an engineering perspective it makes sense to have a "generalist model" underneath that is capable of "taking its best guess" if commanded, and then trying to figure out how sure it is about its guess, build guiderails, etc. Rather than building a model that is implicitly wishy-washy and always second-guessing itself etc. The history of public usage of AI has basically been that too many guiderails make it useless, not just gemini making japanese pharohs to boost diversity or whatever, but frankly even mundane usage is frustratingly punctuated by "sorry I can't tell you about that, I'm just an AI". And frankly it seems best to just give people the model and then if there's domains where a true/false/null/undefined approach makes sense then you build that as a separate layer/guiderail on top of it.
tsol 3y ago
It isn't designed to know things. It doesn't know what exactly it knows, where it could check before answering. It generates an output, which isn't even the same thing every time. So this again is a problem of not understanding how it functions
- someplaceguy 3y ago
  > It isn't designed to know things. It doesn't know what exactly it knows, where it could check before answering. It generates an output, which isn't even the same thing every time. If an entity can predict the correct answer to a question (with a sufficiently low margin of error), then it knows the answer. However, if the prediction contains too much uncertainty, then the entity should not act like they know the answer. The above is valid for humans and LLMs. So we "just" need to model and train LLMs to take uncertainty into account when generating outputs. Easy, right? :)

dragonwriter 3y ago

> Answering "I don't know" or "I can't answer that" is a perfectly plausible response to a difficult logical problem/question. Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I couldn't solve”. And that’s why LLMs behave the way they do.

someplaceguy 3y ago
> Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I couldn't solve”. Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? Sure, there will be an error rate, but surely you can train an LLM to minimize it?
- dragonwriter 3y ago
  > Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? No, I am saying that they are specific trained to do that, and that the results seen in practice on common real-world LLMs reflect the bias of the specific training they are given for providing concrete answers. > Sure, there will be an error rate, but surely you can train an LLM to minimize it? Giving some answer to a question that cannot be infallibly solved analytically is not necessarily an error. In fact, I would argue that providing useful answers in situations like that is among the motivating use cases for AI. (Whether or not the answers current LLMs provide in these cases are useful is another question, but you miss 100% of the shots you don’t take.)
- pixl97 3y ago
  >Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer? This is highly problematic and highly contextualized statement. Imagine you're an accountant with the piece of information $x. The answer you give for the statement "What is $x" is going to be highly dependent on who is answering the question. For example 1. The CEO asks "What is $x" 2. A regulator at the SEC asks "What is $x 3. Some random individual or member of the press asks "What is $x" An LLM doesn't have the other human motivations a person does when asked questions, pretty much at this point with LLMs there are only one or two 'voices' it hears (system prompt and user messages). Whereas a human will commonly lie and say I don't know, it's somewhat questionable if we want LLMs intentionally lying. In addition human information is quite often compartmentalized to keep secrets which is currently not in vogue with LLMs as we are attempting to make oracles that know everything with them.
  - someplaceguy 3y ago
    > The answer you give for the statement "What is $x" is going to be highly dependent on who is answering the question. I assume you meant asking rather than answering? > An LLM doesn't have the other human motivations a person does when asked questions, pretty much at this point with LLMs there are only one or two 'voices' it hears (system prompt and user messages). Why would LLMs need any motivation besides how they are trained to be helpful and the given prompts? In my experience with ChatGPT 4, it seems to be pretty good at discerning what and how to answer based on the prompts and context alone. > Whereas a human will commonly lie and say I don't know, it's somewhat questionable if we want LLMs intentionally lying. Why did you jump to the conclusion that an LLM answering "I don't know" is lying? I want LLMs to answer "I don't know" when they don't have enough information to provide a true answer. That's not lying, in fact it's the opposite, because the alternative is to hallucinate an answer. Hallucinations are the "lies" in this scenario. > In addition human information is quite often compartmentalized to keep secrets which is currently not in vogue with LLMs as we are attempting to make oracles that know everything with them. I'd rather have an oracle that can discriminate when it doesn't have enough information to provide a true answer and replies "I don't know" in such cases (or sometimes answer like "If I were to guess, then bla bla bla, but I'm not sure about this"), than one which always gives confident but sometimes wrong answers.
paulmd 3y ago
if more guiderails are useful to users then such things will surely emerge. but from an engineering perspective it makes sense to have a "generalist model" underneath that is capable of "taking its best guess" if commanded, and then trying to figure out how sure it is about its guess, build guiderails, etc. Rather than building a model that is implicitly wishy-washy and always second-guessing itself etc. The history of public usage of AI has basically been that too many guiderails make it useless, not just gemini making japanese pharohs to boost diversity or whatever, but frankly even mundane usage is frustratingly punctuated by "sorry I can't tell you about that, I'm just an AI". And frankly it seems best to just give people the model and then if there's domains where a true/false/null/undefined approach makes sense then you build that as a separate layer/guiderail on top of it.

tsol 3y ago

It isn't designed to know things. It doesn't know what exactly it knows, where it could check before answering. It generates an output, which isn't even the same thing every time. So this again is a problem of not understanding how it functions

someplaceguy 3y ago
> It isn't designed to know things. It doesn't know what exactly it knows, where it could check before answering. It generates an output, which isn't even the same thing every time. If an entity can predict the correct answer to a question (with a sufficiently low margin of error), then it knows the answer. However, if the prediction contains too much uncertainty, then the entity should not act like they know the answer. The above is valid for humans and LLMs. So we "just" need to model and train LLMs to take uncertainty into account when generating outputs. Easy, right? :)