6 ms·

I find Meta’s approach to hallucinations delightfully counter intuitive. Basically they (and presumably OpenAI and others): - Extract a snippet of training

I find Meta’s approach to hallucinations delightfully counter intuitive. Basically they (and presumably OpenAI and others): - Extract a snippet of training data. - Generate a factual question about it using Llama 3. - Have Llama 3 generate an answer. - Score the response against the original data. - If incorrect, train the model to recognize and refuse incorrect responses. In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. Instead of teaching the model to recognize what it doesn't know, why not teach it using those same examples? Of course the idea is to "connect the unused uncertainty neuron", which makes sense for out-of-context generalization. But we can at least appreciate why this wasn't an obvious thing to do for generation 1 LLMs.

implmntatio 2y ago
> but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. Hard to buy. If a machine makes a mistake, it's because it was configured wrong or because of wear and tear, solar flares or some quake or some manufacturing defect in a part. If a learning machine makes a mistake, it's because it's learning has not extended it's rule set to cover that matrix/mistake/pattern, yet; and so it includes that mistake/matrix and other mistakes, analyses for patterns and then creates mistakes that fall into that pattern. Later doing that in a rolling release or canine kind of way and even later learning machines will do it all live, synchronous to their concurrent actions. But yeah, thinking about that, I see why ML engineers wouldn't get there from scratch. It's a rhythm, after all, an epiphany about or realization of how ones dog, ones brain works, learned and then coded step by step. And there is, of course the variety of how people learn and "realize". Someone has to show us the work of those savant programmers/engineers I still haven't seen a documentary of.
2-3-7-43-1807 2y ago
> In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. But the answer space for LLMs is infinite and unbounded. So, no effort will be complete and you will always end up with the question of how to deal with uncertainty. But I admit this is a bit of hindsight 20/20.
- fenomas 2y ago
  Karpathy's point in the video is that the models don't need to be exhaustively told what they don't know - they already have a good understanding of the extents of their knowledge. Older models just didn't use that understanding; they answered every question confidently because they'd only been trained on confident answers.
  - 2-3-7-43-1807 2y ago
    i don't think that's what he meant and also don't think that is accurate to say they already have an understanding. i'm not even basing my criticism on the anthropomorphization but on the fact that there will be activation constellation that correlate with uncertainty but you have to train them to channel this into an actual response expressing uncertainty ... only then it makes sense to speak of understanding uncertainty.
    - fenomas 2y ago
      Sorry, I don't follow what you're disagreeing with. I was summarizing what Karpathy talks about in the vicinity of 1:31:00 - where he talks (I assume notionally) about a specific neuron lighting lighting up to indicate uncertainty, and how empirically this turns out probably to be the case. Edit: concretely, we can presume that OpenAI didn't specifically train ChatGPT to know that "Orson Kovacs" isn't a famous person, right? That's all I'm saying here - that they trained it how to say it doesn't know things, and it took care of the rest.
      - 2-3-7-43-1807 2y ago
        i think i misinterpreted your first sentence.
        fenomas 2y ago
        Hrrm, I'm reading back and I may have misinterpreted your first post too. If so apologies!

implmntatio 2y ago

> but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. Hard to buy. If a machine makes a mistake, it's because it was configured wrong or because of wear and tear, solar flares or some quake or some manufacturing defect in a part. If a learning machine makes a mistake, it's because it's learning has not extended it's rule set to cover that matrix/mistake/pattern, yet; and so it includes that mistake/matrix and other mistakes, analyses for patterns and then creates mistakes that fall into that pattern. Later doing that in a rolling release or canine kind of way and even later learning machines will do it all live, synchronous to their concurrent actions. But yeah, thinking about that, I see why ML engineers wouldn't get there from scratch. It's a rhythm, after all, an epiphany about or realization of how ones dog, ones brain works, learned and then coded step by step. And there is, of course the variety of how people learn and "realize". Someone has to show us the work of those savant programmers/engineers I still haven't seen a documentary of.

2-3-7-43-1807 2y ago

> In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. But the answer space for LLMs is infinite and unbounded. So, no effort will be complete and you will always end up with the question of how to deal with uncertainty. But I admit this is a bit of hindsight 20/20.

fenomas 2y ago
Karpathy's point in the video is that the models don't need to be exhaustively told what they don't know - they already have a good understanding of the extents of their knowledge. Older models just didn't use that understanding; they answered every question confidently because they'd only been trained on confident answers.
- 2-3-7-43-1807 2y ago
  i don't think that's what he meant and also don't think that is accurate to say they already have an understanding. i'm not even basing my criticism on the anthropomorphization but on the fact that there will be activation constellation that correlate with uncertainty but you have to train them to channel this into an actual response expressing uncertainty ... only then it makes sense to speak of understanding uncertainty.
  - fenomas 2y ago
    Sorry, I don't follow what you're disagreeing with. I was summarizing what Karpathy talks about in the vicinity of 1:31:00 - where he talks (I assume notionally) about a specific neuron lighting lighting up to indicate uncertainty, and how empirically this turns out probably to be the case. Edit: concretely, we can presume that OpenAI didn't specifically train ChatGPT to know that "Orson Kovacs" isn't a famous person, right? That's all I'm saying here - that they trained it how to say it doesn't know things, and it took care of the rest.
    - 2-3-7-43-1807 2y ago
      i think i misinterpreted your first sentence.
      - fenomas 2y ago
        Hrrm, I'm reading back and I may have misinterpreted your first post too. If so apologies!