4 ms·

> In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural langua

> In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding That is a common misunderstanding. LLMs are not Markov chains, and don't need their training dataset to contain multiple (or any!) occurrences of any particular words to be able to understand what is being said in a sentence. Using the example sentence from the GP, this is what o1-mini replied: > No, inserting two gherkins into your nostrils will not enable you to fly. [...] Flying requires overcoming the force of gravity, which cannot be achieved through such means. I tried a few times and got broadly similar negative answers. This ability to generalize beyond the exact contents of their training datasets is what allows other AIs to e.g. produce images of astronauts on the moon riding horses -- they know what astronauts look like, they know what horses and people riding them look like, and they combine the two to make impossible images like that.

ggm 2y ago
I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.
- cma 2y ago
  Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.
semking 2y ago
I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?
- david-gpu 2y ago
  They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
  - semking 2y ago
    Sorry, I just saw your question. I'm still confused by the comment system here! To answer your question, I've noticed LLMs overfit the patterns in their training data. The performance was poor when asked to "reason" in ways that diverge from what they "learned". Things less frequently mentioned in the datasets were clearly weighted as less important. And the opposite is also true. I've seen bias being amplified. For some reason, I've seen words such as "elevate" or "landscape" being overused. And obviously, I've seen so many factual errors. Search for niche topics/individuals that were not covered in the training data and you'll see incredible hallucinations.
cma 2y ago
GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets."
- semking 2y ago
  Interesting paper and they also claim to have discovered a "simple method to detect overfitting during training without using a holdout set"

ggm 2y ago

I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.

cma 2y ago
Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.

semking 2y ago

I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?

david-gpu 2y ago
They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
- semking 2y ago
  Sorry, I just saw your question. I'm still confused by the comment system here! To answer your question, I've noticed LLMs overfit the patterns in their training data. The performance was poor when asked to "reason" in ways that diverge from what they "learned". Things less frequently mentioned in the datasets were clearly weighted as less important. And the opposite is also true. I've seen bias being amplified. For some reason, I've seen words such as "elevate" or "landscape" being overused. And obviously, I've seen so many factual errors. Search for niche topics/individuals that were not covered in the training data and you'll see incredible hallucinations.

cma 2y ago

GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets."

semking 2y ago
Interesting paper and they also claim to have discovered a "simple method to detect overfitting during training without using a holdout set"