4 ms·

The current generation of ChatGPT hallucinates far more systematically than that. The problem is that it can regurgitate what the random dudes from local accele

The current generation of ChatGPT hallucinates far more systematically than that. The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts. We all know blog posts are often a sales pitch and don't include useful advice, because you can charge for the useful advice you give.

tossandthrow 2y ago
> The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts. This is not how modern training pipelines work. The models are trained on proprietary alignments that ensure factuality. Regardless, that random dude from a local accelerator probably regurgitates those blog articles himself.
- yencabulator 2y ago
  "ensures", really?
  - tossandthrow 2y ago
    Like with so many other thing, it is a process. Just like reviews "ensure" correctness on Wikipedia - which is actually a good case. In my childhood teachers would make us avoid Wikipedia. Now they advice students to use it.
    - Jensson 2y ago
      LLMs does compose answers from its corpus, it doesn't refer to it, so it doesn't ensure factuality at all since it can still compose questions in stupid ways just like it always did. Most of the wrong thing it says is just invented by the model, not things it has seen, cleaning up the input doesn't add much there since it writes the same kind of bullshit as bullshitting humans do.
      - tossandthrow 2y ago
        Yes, that is correct, and was also more than indicated in my initial comment. > The software engineer needs to "de-hallucinate" him in order to get a proper answer. The question is not whether a model is sound and complete. The question how LLMs perform in comparison to the existing solutions. Don't let perfect get in the way of good.
        oxfordmale 2y ago
        With humans, it is often easy to establish who hallucinates and who doesn't. We can all identify a snake oil sales pitch. However, with LLM, it can often be a lot harder. Let's take an example. I studied physics and can likely answer most questions on nuclear physics. However, as it is a while back, my answers will not sound as smooth, and I may have to correct myself a few times. If you ask ChatGPT the same questions, it will come up with elegant, convincing answers. As a layman, you will likely prefer what ChatGPT comes up with, as it is answers are eloquent. Of course, until you build a nuclear reactor using ChatGPT knowledge. Once the fuel rods become critical and start making its way through the earth's crust, you may realise you should have relied on my stumbling answers instead.

tossandthrow 2y ago

> The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts. This is not how modern training pipelines work. The models are trained on proprietary alignments that ensure factuality. Regardless, that random dude from a local accelerator probably regurgitates those blog articles himself.

yencabulator 2y ago
"ensures", really?
- tossandthrow 2y ago
  Like with so many other thing, it is a process. Just like reviews "ensure" correctness on Wikipedia - which is actually a good case. In my childhood teachers would make us avoid Wikipedia. Now they advice students to use it.
  - Jensson 2y ago
    LLMs does compose answers from its corpus, it doesn't refer to it, so it doesn't ensure factuality at all since it can still compose questions in stupid ways just like it always did. Most of the wrong thing it says is just invented by the model, not things it has seen, cleaning up the input doesn't add much there since it writes the same kind of bullshit as bullshitting humans do.
    - tossandthrow 2y ago
      Yes, that is correct, and was also more than indicated in my initial comment. > The software engineer needs to "de-hallucinate" him in order to get a proper answer. The question is not whether a model is sound and complete. The question how LLMs perform in comparison to the existing solutions. Don't let perfect get in the way of good.
      - oxfordmale 2y ago
        With humans, it is often easy to establish who hallucinates and who doesn't. We can all identify a snake oil sales pitch. However, with LLM, it can often be a lot harder. Let's take an example. I studied physics and can likely answer most questions on nuclear physics. However, as it is a while back, my answers will not sound as smooth, and I may have to correct myself a few times. If you ask ChatGPT the same questions, it will come up with elegant, convincing answers. As a layman, you will likely prefer what ChatGPT comes up with, as it is answers are eloquent. Of course, until you build a nuclear reactor using ChatGPT knowledge. Once the fuel rods become critical and start making its way through the earth's crust, you may realise you should have relied on my stumbling answers instead.