3 ms·

I've had Gemini offer me words which presumed the search question expected an answer YES, when in fact it's the form of question which expects the answer NO. A

by ggm 2y ago

I've had Gemini offer me words which presumed the search question expected an answer YES, when in fact it's the form of question which expects the answer NO. A fictitious case might be "Is it possible to fly by inserting two gherkins in your nostrils" and the response from Gemini has been in the spirit of "by choosing gherkins with appropriate thrust, it is indeed possible to fly by nasal insertion" It's great at the syntactic parse. It's really not great at the semantic intent sometimes. Don't eat tide pod kids.

semking 2y ago
Interesting example, did this happen directly within Google Search or were you using the gemini[.]google[.]com subdomain? My understanding is that Gemini, like most LLMs, uses probabilistic models to predict/guess the most likely words in a given semantical context. But this logic relies not only on the algorithms but also on the datasets. In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding (NLU). Unsure I'm expressing myself clearly...
- david-gpu 2y ago
  > In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding That is a common misunderstanding. LLMs are not Markov chains, and don't need their training dataset to contain multiple (or any!) occurrences of any particular words to be able to understand what is being said in a sentence. Using the example sentence from the GP, this is what o1-mini replied: > No, inserting two gherkins into your nostrils will not enable you to fly. [...] Flying requires overcoming the force of gravity, which cannot be achieved through such means. I tried a few times and got broadly similar negative answers. This ability to generalize beyond the exact contents of their training datasets is what allows other AIs to e.g. produce images of astronauts on the moon riding horses -- they know what astronauts look like, they know what horses and people riding them look like, and they combine the two to make impossible images like that.
  - ggm 2y ago
    I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.
    - cma 2y ago
      Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.
  - semking 2y ago
    I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?
    - david-gpu 2y ago
      They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
- wongarsu 2y ago
  We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told them that in a month they have a test that consists of completing sentences from the book. Who do you think would do better: the one who tries to remember every sentence, or the one who understands the book's contents? In LLMs we "just" do back propagation and let the model figure out how to do the prediction, and larger models develop something that looks and feels like a degree of understanding and reasoning (incomplete as it may be). Case in point, GPT4o's answer to the above question is: "Inserting two gherkins in your nostrils will not enable you to fly. Flight, as we generally understand it, is achieved through specific principles of physics and engineering, such as those utilized by birds, airplanes, and other flying devices. These principles typically involve lift, thrust, weight, and drag. Gherkins, unfortunately, do not possess any properties that could facilitate flight in a human. If you're interested in flying, consider looking into aviation technology or learning to pilot an aircraft!"
  - semking 2y ago
    Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in the real world, what is the added value of back propagation beyond the obvious examples? There's no creativity whatsoever... we're just working with what already exists.
    - wongarsu 2y ago
      The biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.
- ggm 2y ago
  Search. I think it's a modifying space: what one sees last week may not be the same as what one sees this week, nor need the same query return the same response.
  - semking 2y ago
    Yes, things change. The world changes. Search changes. But let's limit ourselves to Search. Why does Search change? There are new entries and new entities competing for a given search query. There are algorithmic updates that artificially change rankings and how results are displayed. You should read my comment in TechRepublic regarding Google's artificial search experience that just ended last week. Google Makes Changes to Search Features to Appease EU Regulators - Tech Republic (November 27, 2024)
jvm___ 2y ago
"no soot in the pyramids" There is no soot in the corridors and tombs of the pyramids because they were lit by electricity, not flames.
- deleted 2y ago
  [deleted]
- semking 2y ago
  I'm pretty sure the pyramids were powered by nuclear fusion, something we should master in the coming 100 years! :D /S
- areyousure 2y ago
  For those who didn't understand, this is a search result and its AI response: https://www.google.com/search?q=no%20soot%20in%20the%20pyramids https://www.google.com/search?q=no%20soot%20in%20the%20pyram...
  - abenga 2y ago
    Interestingly, you can follow the source links and find the source[1]. I guess Gemini has no way of separating crackpot nonsense from actual knowledge. [1] https://dergipark.org.tr/tr/download/article-file/620376#:~:text=Although%20soot%20was%20found%20in,be%20used%20in%20these%20areas https://dergipark.org.tr/tr/download/article-file/620376#:~:....
deleted 2y ago
[deleted]

by ggm 2y ago

semking 2y ago

Interesting example, did this happen directly within Google Search or were you using the gemini[.]google[.]com subdomain? My understanding is that Gemini, like most LLMs, uses probabilistic models to predict/guess the most likely words in a given semantical context. But this logic relies not only on the algorithms but also on the datasets. In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding (NLU). Unsure I'm expressing myself clearly...

david-gpu 2y ago
> In other words, the datasets would have to contain multiple occurrences of the various very unlikely words you wrote for the model to be decent natural language understanding That is a common misunderstanding. LLMs are not Markov chains, and don't need their training dataset to contain multiple (or any!) occurrences of any particular words to be able to understand what is being said in a sentence. Using the example sentence from the GP, this is what o1-mini replied: > No, inserting two gherkins into your nostrils will not enable you to fly. [...] Flying requires overcoming the force of gravity, which cannot be achieved through such means. I tried a few times and got broadly similar negative answers. This ability to generalize beyond the exact contents of their training datasets is what allows other AIs to e.g. produce images of astronauts on the moon riding horses -- they know what astronauts look like, they know what horses and people riding them look like, and they combine the two to make impossible images like that.
- ggm 2y ago
  I certainly didn't mean literally the gherkin example. It was a fictitious example of the kind of question which has had surprising results. If you have high confidence this either cannot happen, or hasn't happened to you then for your question forms this is good. I am concerned that having seen it, even if I now cannot recreate it, the risk exists.
  - cma 2y ago
    Here's a better example: a common riddle they are trained on, but the prompter changed the genders to something that doesn't puzzle people and write models still assume it is a puzzle, presumably from text statistics, and get it wrong. https://www.reddit.com/r/singularity/comments/1hek6ai/everyone_says_theres_something_different_about/ https://www.reddit.com/r/singularity/comments/1hek6ai/everyo... But Claude nails it. There is a lot more than base LLM training involved since RLHF and other techniques were added, but there are still problems like sycophancy and these kind of crafted misdirections that aren't misdirecrions. Or Claude might have just been trained on someone else doing the same trick to show limitations of LLMs, the internet is vast and I've seen similar experiments it might have seen since it is more recent.
- semking 2y ago
  I think I see what you mean and I should have been more precise. I'll rephrase: most LLMs based on transformer architectures do not rely on the simplistic state transition logic of Markov chains... BUT, while they can use alternative mechanisms such as relative weight (attention), embeddings... they STILL require the pre-existence of exact words, synonyms, or similar/related words in the datasets, correct?
  - david-gpu 2y ago
    They do not require instances of exact words in the training set, either. They can cope with e.g. typos or novel portmanteaus of your own making. Can you think of an example of the limitations you are thinking of?
wongarsu 2y ago
We train them to predict words, but that doesn't mean they have to predict them in a naive statistical way. Imagine you gave a human a physics textbook and told them that in a month they have a test that consists of completing sentences from the book. Who do you think would do better: the one who tries to remember every sentence, or the one who understands the book's contents? In LLMs we "just" do back propagation and let the model figure out how to do the prediction, and larger models develop something that looks and feels like a degree of understanding and reasoning (incomplete as it may be). Case in point, GPT4o's answer to the above question is: "Inserting two gherkins in your nostrils will not enable you to fly. Flight, as we generally understand it, is achieved through specific principles of physics and engineering, such as those utilized by birds, airplanes, and other flying devices. These principles typically involve lift, thrust, weight, and drag. Gherkins, unfortunately, do not possess any properties that could facilitate flight in a human. If you're interested in flying, consider looking into aviation technology or learning to pilot an aircraft!"
- semking 2y ago
  Your example of Memory vs Understanding is crazy... it gave me a counterintuitive and insightful idea that's amazing —and I'm not making fun of you! :D But in the real world, what is the added value of back propagation beyond the obvious examples? There's no creativity whatsoever... we're just working with what already exists.
  - wongarsu 2y ago
    The biggest added value is that it works. Before LLMs we tried all kinds of stuff, including purely statistical methods, and anything more complex than scripted responses was incredibly hard to keep coherent for more than five sentences. GPT2 had similar issues, but modern LLMs are incredibly good. But ultimately the goal are models that have a decent world model and can reason about it. A model that you can ask for a python program that does something completely novel, and you get a working result. We are somewhere on that road, far beyond what we were able to do with statistical methods but still not nearly as good as we wish.
ggm 2y ago
Search. I think it's a modifying space: what one sees last week may not be the same as what one sees this week, nor need the same query return the same response.
- semking 2y ago
  Yes, things change. The world changes. Search changes. But let's limit ourselves to Search. Why does Search change? There are new entries and new entities competing for a given search query. There are algorithmic updates that artificially change rankings and how results are displayed. You should read my comment in TechRepublic regarding Google's artificial search experience that just ended last week. Google Makes Changes to Search Features to Appease EU Regulators - Tech Republic (November 27, 2024)

jvm___ 2y ago

"no soot in the pyramids" There is no soot in the corridors and tombs of the pyramids because they were lit by electricity, not flames.

deleted 2y ago
[deleted]
semking 2y ago
I'm pretty sure the pyramids were powered by nuclear fusion, something we should master in the coming 100 years! :D /S
areyousure 2y ago
For those who didn't understand, this is a search result and its AI response: https://www.google.com/search?q=no%20soot%20in%20the%20pyramids https://www.google.com/search?q=no%20soot%20in%20the%20pyram...
- abenga 2y ago
  Interestingly, you can follow the source links and find the source[1]. I guess Gemini has no way of separating crackpot nonsense from actual knowledge. [1] https://dergipark.org.tr/tr/download/article-file/620376#:~:text=Although%20soot%20was%20found%20in,be%20used%20in%20these%20areas https://dergipark.org.tr/tr/download/article-file/620376#:~:....

deleted 2y ago

[deleted]