4 ms·

I am aware, but think about right after a smaller is done training. The researchers can then quiz it to get a sense of the depth of knowledge it can reliably ci

I am aware, but think about right after a smaller is done training. The researchers can then quiz it to get a sense of the depth of knowledge it can reliably cite, then fine-tune with examples of questions beyond the known depth of the model being refused with "Sorry, I'm a small model and don't have enough info to answer that confidently." Obviously it's asking for a lot to try to cram more "self awareness" into small models, but I doubt the current state of the art is a hard ceiling.

porridgeraisin 1y ago
> then fine-tune with examples of questions beyond the known depth of the model being refused with "Sorry, I'm a small model and don't have enough info to answer that confidently." This has already been tried, llama pioneered it (as far as I can infer from public knowledge, maybe openai did it years ago I don't know). They looped through a bunch of wikipedia pages, made questions out of the info given there, posed them to the LLM and then whenever the answer did not match what was in wikipedia, they went ahead and finetuned on "that question: Sorry I don't know ...". Then, we went one step ahead, and finetuned it to use search in these cases instead of saying I don't know. Finetune it on the answer toolCall("search", "that question", ...) or whatever. Something close to the above is how all models with search tool capability are fine tuned. All these hallucinations are despite those efforts, it was much worse before. This whole method depends on the assumption that there is actually a path in the internal representation that fires when it's gonna hallucinate. The results so far tell us that it is partially true. No way to quantify it of course.
- hnuser123456 1y ago
  Do you have any links on tracing a NN for a hallucination/unconfidence neuron? I do worry that it's possible there isn't an obvious neuron that always goes to 1 when bullshitting, but maybe with the right finetuning, we could induce one?
  - porridgeraisin 1y ago
    Nope, I haven't come across a work that does that. But I also haven't looked in a couple months.

porridgeraisin 1y ago

> then fine-tune with examples of questions beyond the known depth of the model being refused with "Sorry, I'm a small model and don't have enough info to answer that confidently." This has already been tried, llama pioneered it (as far as I can infer from public knowledge, maybe openai did it years ago I don't know). They looped through a bunch of wikipedia pages, made questions out of the info given there, posed them to the LLM and then whenever the answer did not match what was in wikipedia, they went ahead and finetuned on "that question: Sorry I don't know ...". Then, we went one step ahead, and finetuned it to use search in these cases instead of saying I don't know. Finetune it on the answer toolCall("search", "that question", ...) or whatever. Something close to the above is how all models with search tool capability are fine tuned. All these hallucinations are despite those efforts, it was much worse before. This whole method depends on the assumption that there is actually a path in the internal representation that fires when it's gonna hallucinate. The results so far tell us that it is partially true. No way to quantify it of course.

hnuser123456 1y ago
Do you have any links on tracing a NN for a hallucination/unconfidence neuron? I do worry that it's possible there isn't an obvious neuron that always goes to 1 when bullshitting, but maybe with the right finetuning, we could induce one?
- porridgeraisin 1y ago
  Nope, I haven't come across a work that does that. But I also haven't looked in a couple months.