3 ms·

> As an AI assistant, I should be more careful I hate this kind of thing so much.

magicalhippo 2y ago
I've been playing with Gemma locally, and I've had some success by telling it to answer "I don't know" if it doesn't know the answer, or similar escape hatches. Feels like they were trained with a gun to their heads. If I don't tell it it doesn't have to answer it'll generate nonsense in a confident voice.
- ithkuil 2y ago
  The models weights are tuned towards the direction that would cause the model to best fit the training set. It turns out that this process makes it useful at producing mostly sensible predictions (generate output) for text that is not present in the training set (generalization). The reason that works is because there are a lot of patterns and redundancy in the stuff that we feed to the models and the stuff that we ask the models so there is a good chance that interpolating between words and higher level semantics relationship between sentences will make sense quite often. However that doesn't work all the time. And when it doesn't, current models have no way to tell they "don't know". The whole point was to let them generalize beyond the training set and interpolate in order to make decent guesses. There is a lot of research in making models actually reason.
  - magicalhippo 2y ago
    In the Physics of Language Models talk[1], he argues that the model knows it has made a mistake, sometimes even before it has made it. Though apparently training is crucial to make the model be able to use this constructively. That being said, I'm aware that the model doesn't reason in the classical sense. Yet, as I mentioned, it does give me less confabulation when I tell it it's ok not to answer. I will note that when I've tried the same kind of prompts with Phi 3 instruct, it's way worse than Gemma. Though I'm not sure if that's just because of a weak instruction tuning or the underlying training as well, as it frequently ignores parts of my instructions. [1]: https://www.youtube.com/watch?v=yBL7J0kgldU https://www.youtube.com/watch?v=yBL7J0kgldU
    - ithkuil 2y ago
      There are different ways to be wrong. For example you can confabulate "facts" or you can make logical or coherence mistakes. Current LLMs are encouraged to be creative and effectively "make up facts". That's what created the first wow factor. The models are able to write a Star Trek fan fiction model in the style of Shakespeare. They are able to take a poorly written email and make it "sound" better (for some definition of better, e.g. more formal, less formal etc). But then, human psychology kicked in and as soon as you have something that can talk like a human and some marketing folks label as "AI" you start expecting it to be useful also for other tasks, some of which require factual knowledge. Now, it's in theory possible to have a system that you can converse with which can _also_ search and verify knowledge. My point is that this is not the place where LLMs start from. You have to add stuff on top of them (and people are actively researching that)
mdp2021 2y ago
Absolutely. Behaviour that in normal life in clean societies would be "eliciting violence": automated hypocritical lying, apologizing in form and not in substance, making statements based on fictional value instead of truthfulness...
- okasaki 2y ago
  What?
  - mdp2021 2y ago
    What "what"? Did the rest of the comments clarify the points to you or should I formulate a "I am so sorry and heartbroken about having suggested that to play a sound you should use the, as you now inform me, non existing command and parameter `oboe --weird-format mysound.snd`, I'll check my information more thoroughly next time and make sure it will not happen again"...
    - okasaki 2y ago
      I mean I'm just not sure what you're saying. What does "eliciting violence" and "clean society" mean? Are you ok? When a web site says "Sorry, page not found" do you start punching your monitor? When the delivery guy leaves a note saying "Sorry we missed you" do you go to the depot to beat up the employees?
      - mdp2021 2y ago
        > What does "eliciting violence" and "clean society" mean I think you are on a good trail to having understood what they meant. The use of 'sorry' is not generally a problem because it is normally framed within expected behaviour and it can be taken as adequate for a true representation, or not blatantly false. But you could imagine scenarios in which the term would be misused into inappropriate formality or manipulation and yes, disrespect is "eliciting violence". You normally work a way in the situation to avoid violence - that is another story. In "sorry, page not found" 'sorry' is the descriptor for a state (i.e. "not the better case"); in "sorry we missed you" it is just courtesy - and it does not generally cover fault or negligence. But look: there are regions that adopt "your call is important to us", and regions that tend to avoid it - because the suspect of it being inappropriate (false) can be strong. The outputs of LLMs I have used frequently passes the threshold, and possibly their structural engineering - if you had in front of you a worker, in flesh and bones, that in its outputs wrote plausible fiction ("I imagined a command `oboe` because it sounded good in the story") as opposed to answering your question, but under the veneer of answering questions (which implies, outputting relevant world assessments, Truth based), that would be a right "sore" for "sorry". The anthropomorphic features of LLMs compromise the quality of their outputs in terms of form, especially in solution-finding attempts that become loops of "This is the solution" // "Are you sure?" // "Definitely" // "It is not" // "Oh, I'm so sorry! It will not happen again. This is the solution" (loop...). Edit: it seems you may have also asked for clarifications about the contextual expression «clean societies». Those societies cybernetically healthy, in which feedback mechanisms work properly to fine-tune general mechanisms - with particular regard to fixing individual, then collective behaviour.

I've been playing with Gemma locally, and I've had some success by telling it to answer "I don't know" if it doesn't know the answer, or similar escape hatches. Feels like they were trained with a gun to their heads. If I don't tell it it doesn't have to answer it'll generate nonsense in a confident voice.

ithkuil 2y ago
The models weights are tuned towards the direction that would cause the model to best fit the training set. It turns out that this process makes it useful at producing mostly sensible predictions (generate output) for text that is not present in the training set (generalization). The reason that works is because there are a lot of patterns and redundancy in the stuff that we feed to the models and the stuff that we ask the models so there is a good chance that interpolating between words and higher level semantics relationship between sentences will make sense quite often. However that doesn't work all the time. And when it doesn't, current models have no way to tell they "don't know". The whole point was to let them generalize beyond the training set and interpolate in order to make decent guesses. There is a lot of research in making models actually reason.
- magicalhippo 2y ago
  In the Physics of Language Models talk[1], he argues that the model knows it has made a mistake, sometimes even before it has made it. Though apparently training is crucial to make the model be able to use this constructively. That being said, I'm aware that the model doesn't reason in the classical sense. Yet, as I mentioned, it does give me less confabulation when I tell it it's ok not to answer. I will note that when I've tried the same kind of prompts with Phi 3 instruct, it's way worse than Gemma. Though I'm not sure if that's just because of a weak instruction tuning or the underlying training as well, as it frequently ignores parts of my instructions. [1]: https://www.youtube.com/watch?v=yBL7J0kgldU https://www.youtube.com/watch?v=yBL7J0kgldU
  - ithkuil 2y ago
    There are different ways to be wrong. For example you can confabulate "facts" or you can make logical or coherence mistakes. Current LLMs are encouraged to be creative and effectively "make up facts". That's what created the first wow factor. The models are able to write a Star Trek fan fiction model in the style of Shakespeare. They are able to take a poorly written email and make it "sound" better (for some definition of better, e.g. more formal, less formal etc). But then, human psychology kicked in and as soon as you have something that can talk like a human and some marketing folks label as "AI" you start expecting it to be useful also for other tasks, some of which require factual knowledge. Now, it's in theory possible to have a system that you can converse with which can _also_ search and verify knowledge. My point is that this is not the place where LLMs start from. You have to add stuff on top of them (and people are actively researching that)

mdp2021 2y ago

Absolutely. Behaviour that in normal life in clean societies would be "eliciting violence": automated hypocritical lying, apologizing in form and not in substance, making statements based on fictional value instead of truthfulness...

okasaki 2y ago
What?
- mdp2021 2y ago
  What "what"? Did the rest of the comments clarify the points to you or should I formulate a "I am so sorry and heartbroken about having suggested that to play a sound you should use the, as you now inform me, non existing command and parameter `oboe --weird-format mysound.snd`, I'll check my information more thoroughly next time and make sure it will not happen again"...
  - okasaki 2y ago
    I mean I'm just not sure what you're saying. What does "eliciting violence" and "clean society" mean? Are you ok? When a web site says "Sorry, page not found" do you start punching your monitor? When the delivery guy leaves a note saying "Sorry we missed you" do you go to the depot to beat up the employees?
    - mdp2021 2y ago
      > What does "eliciting violence" and "clean society" mean I think you are on a good trail to having understood what they meant. The use of 'sorry' is not generally a problem because it is normally framed within expected behaviour and it can be taken as adequate for a true representation, or not blatantly false. But you could imagine scenarios in which the term would be misused into inappropriate formality or manipulation and yes, disrespect is "eliciting violence". You normally work a way in the situation to avoid violence - that is another story. In "sorry, page not found" 'sorry' is the descriptor for a state (i.e. "not the better case"); in "sorry we missed you" it is just courtesy - and it does not generally cover fault or negligence. But look: there are regions that adopt "your call is important to us", and regions that tend to avoid it - because the suspect of it being inappropriate (false) can be strong. The outputs of LLMs I have used frequently passes the threshold, and possibly their structural engineering - if you had in front of you a worker, in flesh and bones, that in its outputs wrote plausible fiction ("I imagined a command `oboe` because it sounded good in the story") as opposed to answering your question, but under the veneer of answering questions (which implies, outputting relevant world assessments, Truth based), that would be a right "sore" for "sorry". The anthropomorphic features of LLMs compromise the quality of their outputs in terms of form, especially in solution-finding attempts that become loops of "This is the solution" // "Are you sure?" // "Definitely" // "It is not" // "Oh, I'm so sorry! It will not happen again. This is the solution" (loop...). Edit: it seems you may have also asked for clarifications about the contextual expression «clean societies». Those societies cybernetically healthy, in which feedback mechanisms work properly to fine-tune general mechanisms - with particular regard to fixing individual, then collective behaviour.