3 ms·

If a person is pretending well enough, it appears genuine. Why would a model learn something it has no information about? It would just learn to be nice, which

If a person is pretending well enough, it appears genuine. Why would a model learn something it has no information about? It would just learn to be nice, which largely seems to be the case for base models. It's part of the "Can of seltzer" problem as I've heard it called. I can describe and write down what I'm seeing on my desk, where there is a can of seltzer I open and drink. A model trained on this description has a fundamental disconnect with reality, because continuing that text requires you to know more information than is contained in the text. It doesn't learn anything about the world, it learns that sometimes in a text there is a can of seltzer, and you can list random things when describing a desk. It's probably why LLMs make so much shit up, because they have no real point of reference and their world view is composed of layers of things that appear nonsensical and self contradictory because they are trained on piles of writings that are hopelessly out of context. The fact that they are even coherent and semi-reliable is a downright miracle.

mikrl 2y ago
>If a person is pretending well enough, it appears genuine There are multiple readings to most statements. “I loved the restaurant we went to” Could be parsed in at least 3 ways without any additional context. Sincere, sarcastic implying the restaurant was bad, indignant implying another restaurant was worse. Without an alignment goal like “always assume sincerity” which itself can backfire, how can you control what an LLM generates? There is no universally derivable law saying any particular interpretation is the right one. This doesn’t even begin to touch on how there may be a signal too weak for humans to perceive, but an LLM could focus on, leading to many other wild and wonderful interpretations mined from the data.
- moffkalast 2y ago
  Well there is the only law that goes with deep learning, the law of statistics. The interpretation that has the highest number of occurrences will most likely be preferred. If you take the standard dataset, i.e. the internet, I would suppose it's actually not that morally bad on average because the sites it's sourced from are largely moderated, and the ones that aren't tend to be thrown out. So there would have to be an inherent lawful, positive bias due to the general lack of the opposite in the ground truth examples provided.

mikrl 2y ago

>If a person is pretending well enough, it appears genuine There are multiple readings to most statements. “I loved the restaurant we went to” Could be parsed in at least 3 ways without any additional context. Sincere, sarcastic implying the restaurant was bad, indignant implying another restaurant was worse. Without an alignment goal like “always assume sincerity” which itself can backfire, how can you control what an LLM generates? There is no universally derivable law saying any particular interpretation is the right one. This doesn’t even begin to touch on how there may be a signal too weak for humans to perceive, but an LLM could focus on, leading to many other wild and wonderful interpretations mined from the data.

moffkalast 2y ago
Well there is the only law that goes with deep learning, the law of statistics. The interpretation that has the highest number of occurrences will most likely be preferred. If you take the standard dataset, i.e. the internet, I would suppose it's actually not that morally bad on average because the sites it's sourced from are largely moderated, and the ones that aren't tend to be thrown out. So there would have to be an inherent lawful, positive bias due to the general lack of the opposite in the ground truth examples provided.