5 ms·

Interesting example, did this happen directly within Google Search or were you using the gemini[.]google[.]com subdomain? My understanding is that Gemini, like

Interesting example, did this happen directly within Google Search or were you using the gemini[.]google[.]com subdomain? My understanding is that Gemini, like most LLMs, uses probabilistic models to predict/guess the most likely words in a given semantical context. But this logic relies not only on the algorithms but also on the datasets. In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding (NLU). Unsure I'm expressing myself clearly...

david-gpu 2y ago
> In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding That is a common misunderstanding. LLMs are not Markov chains, and don't need their training dataset to contain multiple (or any!) occurrences of any particular words to be able to understand what is being said in a sentence. Using the example sentence from the GP, this is what o1-mini replied: > No, inserting two gherkins into your nostrils will not enable you to fly. [...] Flying requires overcoming the force of gravity, which cannot be achieved through such means. I tried a few times and got broadly similar negative answers. This ability to generalize beyond the exact contents of their training datasets is what allows other AIs to e.g. produce images of astronauts on the moon riding horses -- they know what astronauts look like, they know what horses and people riding them look like, and they combine the two to make impossible images like that.
- ggm 2y ago
  I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.
  - cma 2y ago
    Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.
- semking 2y ago
  I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?
  - david-gpu 2y ago
    They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
    - semking 2y ago
      Sorry, I just saw your question. I'm still confused by the comment system here! To answer your question, I've noticed LLMs overfit the patterns in their training data. The performance was poor when asked to "reason" in ways that diverge from what they "learned". Things less frequently mentioned in the datasets were clearly weighted as less important. And the opposite is also true. I've seen bias being amplified. For some reason, I've seen words such as "elevate" or "landscape" being overused. And obviously, I've seen so many factual errors. Search for niche topics/individuals that were not covered in the training data and you'll see incredible hallucinations.
- cma 2y ago
  GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets."
  - semking 2y ago
    Interesting paper and they also claim to have discovered a "simple method to detect overfitting during training without using a holdout set"
wongarsu 2y ago
We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told them that in a month they have a test that consists of completing sentences from the book. Who do you think would do better: the one who tries to remember every sentence, or the one who understands the book's contents? In LLMs we "just" do back propagation and let the model figure out how to do the prediction, and larger models develop something that looks and feels like a degree of understanding and reasoning (incomplete as it may be). Case in point, GPT4o's answer to the above question is: "Inserting two gherkins in your nostrils will not enable you to fly. Flight, as we generally understand it, is achieved through specific principles of physics and engineering, such as those utilized by birds, airplanes, and other flying devices. These principles typically involve lift, thrust, weight, and drag. Gherkins, unfortunately, do not possess any properties that could facilitate flight in a human. If you're interested in flying, consider looking into aviation technology or learning to pilot an aircraft!"
- semking 2y ago
  Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in the real world, what is the added value of back propagation beyond the obvious examples? There's no creativity whatsoever... we're just working with what already exists.
  - wongarsu 2y ago
    The biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.
ggm 2y ago
Search. I think it's a modifying space: what one sees last week may not be the same as what one sees this week, nor need the same query return the same response.
- semking 2y ago
  Yes, things change. The world changes. Search changes. But let's limit ourselves to Search. Why does Search change? There are new entries and new entities competing for a given search query. There are algorithmic updates that artificially change rankings and how results are displayed. You should read my comment in TechRepublic regarding Google's artificial search experience that just ended last week. Google Makes Changes to Search Features to Appease EU Regulators - Tech Republic (November 27, 2024)

david-gpu 2y ago

> In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding That is a common misunderstanding. LLMs are not Markov chains, and don't need their training dataset to contain multiple (or any!) occurrences of any particular words to be able to understand what is being said in a sentence. Using the example sentence from the GP, this is what o1-mini replied: > No, inserting two gherkins into your nostrils will not enable you to fly. [...] Flying requires overcoming the force of gravity, which cannot be achieved through such means. I tried a few times and got broadly similar negative answers. This ability to generalize beyond the exact contents of their training datasets is what allows other AIs to e.g. produce images of astronauts on the moon riding horses -- they know what astronauts look like, they know what horses and people riding them look like, and they combine the two to make impossible images like that.

ggm 2y ago
I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.
- cma 2y ago
  Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.
semking 2y ago
I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?
- david-gpu 2y ago
  They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
  - semking 2y ago
    Sorry, I just saw your question. I'm still confused by the comment system here! To answer your question, I've noticed LLMs overfit the patterns in their training data. The performance was poor when asked to "reason" in ways that diverge from what they "learned". Things less frequently mentioned in the datasets were clearly weighted as less important. And the opposite is also true. I've seen bias being amplified. For some reason, I've seen words such as "elevate" or "landscape" being overused. And obviously, I've seen so many factual errors. Search for niche topics/individuals that were not covered in the training data and you'll see incredible hallucinations.
cma 2y ago
GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets."
- semking 2y ago
  Interesting paper and they also claim to have discovered a "simple method to detect overfitting during training without using a holdout set"

wongarsu 2y ago

We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told them that in a month they have a test that consists of completing sentences from the book. Who do you think would do better: the one who tries to remember every sentence, or the one who understands the book's contents? In LLMs we "just" do back propagation and let the model figure out how to do the prediction, and larger models develop something that looks and feels like a degree of understanding and reasoning (incomplete as it may be). Case in point, GPT4o's answer to the above question is: "Inserting two gherkins in your nostrils will not enable you to fly. Flight, as we generally understand it, is achieved through specific principles of physics and engineering, such as those utilized by birds, airplanes, and other flying devices. These principles typically involve lift, thrust, weight, and drag. Gherkins, unfortunately, do not possess any properties that could facilitate flight in a human. If you're interested in flying, consider looking into aviation technology or learning to pilot an aircraft!"

semking 2y ago
Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in the real world, what is the added value of back propagation beyond the obvious examples? There's no creativity whatsoever... we're just working with what already exists.
- wongarsu 2y ago
  The biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.

ggm 2y ago

Search. I think it's a modifying space: what one sees last week may not be the same as what one sees this week, nor need the same query return the same response.

semking 2y ago
Yes, things change. The world changes. Search changes. But let's limit ourselves to Search. Why does Search change? There are new entries and new entities competing for a given search query. There are algorithmic updates that artificially change rankings and how results are displayed. You should read my comment in TechRepublic regarding Google's artificial search experience that just ended last week. Google Makes Changes to Search Features to Appease EU Regulators - Tech Republic (November 27, 2024)