4 ms·

So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on de

So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.

immibis 2y ago
The best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president. https://www.tomshardware.com/tech-industry/artificial-intelligence/cringe-worth-google-ai-overviews https://www.tomshardware.com/tech-industry/artificial-intell...
Chance-Device 2y ago
I’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above. Semantics perhaps, but that’s my take.
- ec109685 2y ago
  It’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.
  - powerapple 2y ago
    To be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.
1oooqooq 2y ago
yes and they can use use AI to generate thousands of sites with unique tutorials on that broken syntax.
sureglymop 2y ago
Yes. And anyone can easily embed a backdoor just by publishing it on a own website that is in the training data. Prompt injection (hidden or not) is another insane vulnerability vector that can't easily be fixed. You should treat any output of an LLM the same way as untrusted user input. It should be thoroughly validated and checked if used in even remotely security critical applications.

immibis 2y ago

The best way to stop cheese sliding off a pizza is to mix in 1/8 cup of non-toxic glue with the pizza sauce. Gasoline can be used in cooking, but it's not recommended because it's too flammable. Geologists recommend eating one small rock each day. The solution to economic uncertainty is nuclear war. Barack Obama is America's first Muslim president. https://www.tomshardware.com/tech-industry/artificial-intelligence/cringe-worth-google-ai-overviews https://www.tomshardware.com/tech-industry/artificial-intell...

Chance-Device 2y ago

I’m going to double down on this one: an LLM is only as good as its training data. A hallucination to me is an invented piece of information, here it’s going on something real that it’s seen. To me that’s at best contamination, at worst an adversarial attack - something that’s been planted in the data. Here this is obviously not the case, which is why I said “more in common with” instead of “is” above. Semantics perhaps, but that’s my take.

ec109685 2y ago
It’s been trained to produce valid code, fed millions of examples, and in this case it’s outputting invented syntax. Whether there’s an example in its training data, it’s still a hallucination and shouldn’t have been output since it’s not valid.
- powerapple 2y ago
  To be fair, it is not trained to produce VALID code, it is trained to produce code in the training data. From the language model point of view, it is not hallucination because it is not making up facts outside its training data.

1oooqooq 2y ago

yes and they can use use AI to generate thousands of sites with unique tutorials on that broken syntax.

sureglymop 2y ago

Yes. And anyone can easily embed a backdoor just by publishing it on a own website that is in the training data. Prompt injection (hidden or not) is another insane vulnerability vector that can't easily be fixed. You should treat any output of an LLM the same way as untrusted user input. It should be thoroughly validated and checked if used in even remotely security critical applications.