5 ms·

So... we’ve got ML models to the point of being able to hallucinate?

yarg 4y ago
Hallucination in context means something rather different. Something like coming to accept some data-set local optima as a global truth and deriving beliefs without underlying basis.
hnfong 4y ago
"Yes"? The whole point of these models is to make up something that seems "realistic", but not regurgitate the original contents. If that's not hallucination I don't know what is.
mattkrause 4y ago
Yup: prompt a large language model to tell you about itself and it will sometimes weave weird tales. For example, I just had the following interaction with GTP2large[0]: Me: What is your name? Model: I'm Mark A. <continue> Dreyer, and I'm with the <continue> Federal Communications Commission (FCC) Obviously, it's not actually a guy named Mark A. Dreyer and, as far as I can tell, there is no Mark. A. Dreyer in a prominent role at the FCC. That, as far as I'm concerned, is essentially a hallucination. If you think about how they work, this isn't totally surprising. The model has been programmed to take the input, turn it into a series of tokens, and then produce another series of tokens that are especially likely to follow the input ones. That's it. To the extent that this corresponds to reality, it's only because reality is overrepresented in the training data, and so token sequences corresponding to reality are often more likely under its model. This also explains how it goes off the rails: Mark is a very common name (in the top 25 for men for decades) and once you generate a stereotypical middle-aged white guy name, it's not a huge leap to him being a bureaucrat, etc. I forgot to delete the huggingface boilerplate, which talks about advances in technology (etc), which probably sent it down the FCC route too.
- mrcartmeneses 4y ago
  That’s totally wild. The visual ML output is very obviously like tripping. But what you’re describing is very similar to how someone once described being high on acid to me, where no sentence could be completed without it turning into another and another and another until it was all just gibberish. It’s an interesting feat to be at this stage simply though back propagating neural nets, but there’s a huge gulf between where we are and creating __understanding__ (he said, trying to sound smart)

yarg 4y ago

Hallucination in context means something rather different. Something like coming to accept some data-set local optima as a global truth and deriving beliefs without underlying basis.

hnfong 4y ago

"Yes"? The whole point of these models is to make up something that seems "realistic", but not regurgitate the original contents. If that's not hallucination I don't know what is.

mattkrause 4y ago

Yup: prompt a large language model to tell you about itself and it will sometimes weave weird tales. For example, I just had the following interaction with GTP2large[0]: Me: What is your name? Model: I'm Mark A. <continue> Dreyer, and I'm with the <continue> Federal Communications Commission (FCC) Obviously, it's not actually a guy named Mark A. Dreyer and, as far as I can tell, there is no Mark. A. Dreyer in a prominent role at the FCC. That, as far as I'm concerned, is essentially a hallucination. If you think about how they work, this isn't totally surprising. The model has been programmed to take the input, turn it into a series of tokens, and then produce another series of tokens that are especially likely to follow the input ones. That's it. To the extent that this corresponds to reality, it's only because reality is overrepresented in the training data, and so token sequences corresponding to reality are often more likely under its model. This also explains how it goes off the rails: Mark is a very common name (in the top 25 for men for decades) and once you generate a stereotypical middle-aged white guy name, it's not a huge leap to him being a bureaucrat, etc. I forgot to delete the huggingface boilerplate, which talks about advances in technology (etc), which probably sent it down the FCC route too.

mrcartmeneses 4y ago
That’s totally wild. The visual ML output is very obviously like tripping. But what you’re describing is very similar to how someone once described being high on acid to me, where no sentence could be completed without it turning into another and another and another until it was all just gibberish. It’s an interesting feat to be at this stage simply though back propagating neural nets, but there’s a huge gulf between where we are and creating __understanding__ (he said, trying to sound smart)