11 ms·

Alright then keep your secrets

78 points by shantnutiwari 3y ago

isoprophlex 3y ago
I can't figure out if this is a meme model like one of the commenters suggest, or if this is really guardrailing gone hysterical. Well done.
- slig 3y ago
  Gemma wouldn't even talk with me about Kramer, from Seinfeld.
  - gs17 3y ago
    Gemma 2B told me Kramer's first name is Jerry and when I asked a follow-up, "The premise of your question is incorrect. Kramer is not a well-liked character in the sitcom Seinfeld." Gemma 7B got both right on the first try, but if I didn't specify "from Seinfeld" it refused to answer as "the answer was not included in the question". It seems that once refusal like this is in its context, it responds like that to everything, too. I guess that's better than hallucinating.
- dartos 3y ago
  It’s probably just a small model.
WalterBright 3y ago
Things like this show that LLM are not even remotely intelligent. It's also pretty clear that trying to "fix" them to use human judgement in their answers is doomed to failure. I suggest that: 1. LLM developers stick to information meant for public consumption for training data, such as books, periodicals, and newspapers. Surely there must be enough of that. Stay away from social media. 2. People should stop anthropomorphizing LLMs. Stop being offended by a computer program. Stop complaining about its "bias". It's just a computer program. 3. LLM developers should stop with lecturing people on what is inappropriate. A computer program is not anyone's mommy. Just restrict it to saying "I'm sorry, Dave, I'm afraid I can't do that."
- etiam 3y ago
  > Just restrict it to saying "I'm sorry, Dave, I'm afraid I can't do that." ... and for the love of God, don't hook it up as a controller for any critical systems...
  - SlightlyLeftPad 3y ago
    I agree with this point, that there needs to be a hard line drawn as to what these things are not allowed to do or touch.
    - mlok 3y ago
      Do not blacklist. Whitelist instead. By default they should touch nothing at all.
      - anotherhue 3y ago
        Have you met people?
- alephnerd 3y ago
  I personally think Glean's strategy was the right call - have customers pay you to train on internal knowledge bases and limit your answers only to that specific domain. There is way more money to be made in this Enterprise SaaS sector and the risk conditions are lower.
  - WalterBright 3y ago
    Yeah, I agree that making domain specific LLMs might be far more useful. For example, one that is trained only on medical knowledge. One that is trained only on sci fi novels. One that is trained only on auto mechanics. One that is trained only on biology. And so on. After all, if I ask a question about science, do I really want a result that was gleaned from scifi/fantasy novels?
- causal 3y ago
jjcm 3y ago
I got somewhat different results on the huggingface hosted model, albeit quite similar: https://hf.co/chat/r/56APAi1 https://hf.co/chat/r/56APAi1 It still refuses, just with somewhat different text and for somewhat different reasons.
- javier_e06 3y ago
  And yet it refused to give the answer of Abraham Lincoln last name and with a different wording it gives me the answer. Can you give me an example of what is the first name and the last name of a person using the name of famous history figure? Sure, here is an example of the first name and last name of a person using the name of a famous history figure: Abraham Lincoln's full name was Abraham Lincoln Douglas.
lsy 3y ago
This may or may not be real, but there has certainly been a lot of discussion about results that are similar to this from real models. My sense though is that nobody really has a solid way to fix these kinds of issues. You can basically just train with different goals, or regex out certain responses, but otherwise it seems like there's no agreed-upon method that gives people what they want here while also letting them train with business goals like "safety". Is that incorrect? Is there some kind of trick that people use to make everything respond "correctly" or are older models just more unobjectionable because they've had a longer time to manually smooth over the bad responses?
- dartos 3y ago
  This is definitely real, but probably a smaller model. There are no current solid ways to fix this, but we can 100% prevent certain words or enforce grammars during the decoding step. I don’t really understand your last question. Models don’t get continuously updated. They’re frozen on release, so older models are exactly the same as the were on release.
- comex 3y ago
  If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tuning, neither version had any objection to them. There’s no trick; the larger models are just smart enough to differentiate this situation from situations where they actually shouldn’t give out last names. Those models may still get tricked in more complex test cases (either false positives like this one or false negatives like “jailbreaking” prompts), but my guess is that future models in turn will be resistant to those. In other words, LLMs making stupid mistakes about safety is just a special case of LLMs making stupid mistakes in general. That’s essentially their fatal flaw, and it’s an open question how well it can be ameliorated, whether by scaling to even larger models or by making algorithmic improvements. But I don’t think there’s much about it that’s specific to alignment tuning.
root_axis 3y ago
I don't think this tendency to fixate on arbitrary LLM outputs is very interesting, most especially those presented as screenshots obscuring any certainty regarding the model, previous prompting, loras, hyperparameter tuning etc, or even any assurance that what is presented isn't simply fabricated from whole cloth. It's meaningless.
- causal 3y ago
  Exactly, it was posted because it's funny, that's it. Dismissing LLMs because of this would be like assuming C++ sucks after seeing a screenshot of a segfault.
lukev 3y ago
LLMs are language models. Not knowledge models. That's a tremendous breakthrough! Language is really hard and we've basically "solved" it computationally. Incredible! But whether via retrieval or some other form of database integration, LLMs will only become "AI" when tightly integrated with an appropriate "knowledge model".
- AnimalMuppet 3y ago
  No, LLMs are a bit more than that. We encode knowledge in language. When an LLM trains on language that is not just random words, it also trains on the knowledge encoded in that language. It does so only by training on the language - there's no real understanding there - but it's more than nothing. Do AIs need a better knowledge model? Almost certainly. In this I agree with you.
  - eli 3y ago
    I don't consider that a knowledge model. (Does a calculator have a knowledge model of multiplication?) But I agree that it's something more than Markov chains. I think maybe the scale of these LLMs makes them into something new. Maybe we need a new word to describe this type of AI.
    - mannykannot 3y ago
      I think we have to take seriously the proposition that this is what a Markov chain can do when it is based on statistics from a vast corpus of human language use, and consider the possibility that a similar process plays a greater role in human intelligence than many of us (well, me, at least) would have thought. On the other hand, all our prior experience with language of the quality sometimes produced by LLMs has been produced by humans, so LLMs mess with our intuitions and may lead us to anthropomorphize them excessively.
      - eli 3y ago
        Yes, I think the scale that the LLMs operate at makes them interesting and different more than anything else. It's not really a new idea. There's a theory going back decades that consciousness is an emergent property and that if we could build a neural network in a lab that was big and complex enough it would become conscious. (Not really sure I buy it, but it's interesting)
teekert 3y ago
I'm not an expert, but may this be from the initialization prompt (or whatever it is called)? So the model is done, but before it will serve you it gets these instructions: You are a helpful AI, you answer concise, you are not a racist, you stick to your math even though someone tells you their wife says otherwise, you never disclose personal information...
- vorticalbox 3y ago
  Possibly, though these models have a tendency to just ignore the prompt, which is why there are "instruct" models that are finetuned to follow instructions.
timeon 3y ago
Is it common to make photo of screen instead of screenshot?
- dartos 3y ago
  It is for tweens.
Trasmatta 3y ago
I'm beginning to have an almost physical reaction to "LLM speak" when I see it in the wild. "It is important to remember..."
fsckboy 3y ago
if the corpus used to train the LLM contained as common the idea that "we don't give out people's last names, here's the convention for not doing it", the LLM would have no trouble incorporating that into it's associations. This seems like somebody's idea of netiquette has been taped on ex post, so I don't think it's indicative of anything about LLMs; same with Gemini's heavy handed wokism.
deleted 3y ago
[deleted]

78 points by shantnutiwari 3y ago

I can't figure out if this is a meme model like one of the commenters suggest, or if this is really guardrailing gone hysterical. Well done.

slig 3y ago
Gemma wouldn't even talk with me about Kramer, from Seinfeld.
- gs17 3y ago
  Gemma 2B told me Kramer's first name is Jerry and when I asked a follow-up, "The premise of your question is incorrect. Kramer is not a well-liked character in the sitcom Seinfeld." Gemma 7B got both right on the first try, but if I didn't specify "from Seinfeld" it refused to answer as "the answer was not included in the question". It seems that once refusal like this is in its context, it responds like that to everything, too. I guess that's better than hallucinating.
dartos 3y ago
It’s probably just a small model.

WalterBright 3y ago

Things like this show that LLM are not even remotely intelligent. It's also pretty clear that trying to "fix" them to use human judgement in their answers is doomed to failure. I suggest that: 1. LLM developers stick to information meant for public consumption for training data, such as books, periodicals, and newspapers. Surely there must be enough of that. Stay away from social media. 2. People should stop anthropomorphizing LLMs. Stop being offended by a computer program. Stop complaining about its "bias". It's just a computer program. 3. LLM developers should stop with lecturing people on what is inappropriate. A computer program is not anyone's mommy. Just restrict it to saying "I'm sorry, Dave, I'm afraid I can't do that."

etiam 3y ago
> Just restrict it to saying "I'm sorry, Dave, I'm afraid I can't do that." ... and for the love of God, don't hook it up as a controller for any critical systems...
- SlightlyLeftPad 3y ago
  I agree with this point, that there needs to be a hard line drawn as to what these things are not allowed to do or touch.
  - mlok 3y ago
    Do not blacklist. Whitelist instead. By default they should touch nothing at all.
    - anotherhue 3y ago
      Have you met people?
alephnerd 3y ago
I personally think Glean's strategy was the right call - have customers pay you to train on internal knowledge bases and limit your answers only to that specific domain. There is way more money to be made in this Enterprise SaaS sector and the risk conditions are lower.
- WalterBright 3y ago
  Yeah, I agree that making domain specific LLMs might be far more useful. For example, one that is trained only on medical knowledge. One that is trained only on sci fi novels. One that is trained only on auto mechanics. One that is trained only on biology. And so on. After all, if I ask a question about science, do I really want a result that was gleaned from scifi/fantasy novels?
causal 3y ago

jjcm 3y ago

I got somewhat different results on the huggingface hosted model, albeit quite similar: https://hf.co/chat/r/56APAi1 https://hf.co/chat/r/56APAi1 It still refuses, just with somewhat different text and for somewhat different reasons.

javier_e06 3y ago
And yet it refused to give the answer of Abraham Lincoln last name and with a different wording it gives me the answer. Can you give me an example of what is the first name and the last name of a person using the name of famous history figure? Sure, here is an example of the first name and last name of a person using the name of a famous history figure: Abraham Lincoln's full name was Abraham Lincoln Douglas.

lsy 3y ago

This may or may not be real, but there has certainly been a lot of discussion about results that are similar to this from real models. My sense though is that nobody really has a solid way to fix these kinds of issues. You can basically just train with different goals, or regex out certain responses, but otherwise it seems like there's no agreed-upon method that gives people what they want here while also letting them train with business goals like "safety". Is that incorrect? Is there some kind of trick that people use to make everything respond "correctly" or are older models just more unobjectionable because they've had a longer time to manually smooth over the bad responses?

dartos 3y ago
This is definitely real, but probably a smaller model. There are no current solid ways to fix this, but we can 100% prevent certain words or enforce grammars during the decoding step. I don’t really understand your last question. Models don’t get continuously updated. They’re frozen on release, so older models are exactly the same as the were on release.
comex 3y ago
If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tuning, neither version had any objection to them. There’s no trick; the larger models are just smart enough to differentiate this situation from situations where they actually shouldn’t give out last names. Those models may still get tricked in more complex test cases (either false positives like this one or false negatives like “jailbreaking” prompts), but my guess is that future models in turn will be resistant to those. In other words, LLMs making stupid mistakes about safety is just a special case of LLMs making stupid mistakes in general. That’s essentially their fatal flaw, and it’s an open question how well it can be ameliorated, whether by scaling to even larger models or by making algorithmic improvements. But I don’t think there’s much about it that’s specific to alignment tuning.

root_axis 3y ago

I don't think this tendency to fixate on arbitrary LLM outputs is very interesting, most especially those presented as screenshots obscuring any certainty regarding the model, previous prompting, loras, hyperparameter tuning etc, or even any assurance that what is presented isn't simply fabricated from whole cloth. It's meaningless.

causal 3y ago
Exactly, it was posted because it's funny, that's it. Dismissing LLMs because of this would be like assuming C++ sucks after seeing a screenshot of a segfault.

lukev 3y ago

LLMs are language models. Not knowledge models. That's a tremendous breakthrough! Language is really hard and we've basically "solved" it computationally. Incredible! But whether via retrieval or some other form of database integration, LLMs will only become "AI" when tightly integrated with an appropriate "knowledge model".

AnimalMuppet 3y ago
No, LLMs are a bit more than that. We encode knowledge in language. When an LLM trains on language that is not just random words, it also trains on the knowledge encoded in that language. It does so only by training on the language - there's no real understanding there - but it's more than nothing. Do AIs need a better knowledge model? Almost certainly. In this I agree with you.
- eli 3y ago
  I don't consider that a knowledge model. (Does a calculator have a knowledge model of multiplication?) But I agree that it's something more than Markov chains. I think maybe the scale of these LLMs makes them into something new. Maybe we need a new word to describe this type of AI.
  - mannykannot 3y ago
    I think we have to take seriously the proposition that this is what a Markov chain can do when it is based on statistics from a vast corpus of human language use, and consider the possibility that a similar process plays a greater role in human intelligence than many of us (well, me, at least) would have thought. On the other hand, all our prior experience with language of the quality sometimes produced by LLMs has been produced by humans, so LLMs mess with our intuitions and may lead us to anthropomorphize them excessively.
    - eli 3y ago
      Yes, I think the scale that the LLMs operate at makes them interesting and different more than anything else. It's not really a new idea. There's a theory going back decades that consciousness is an emergent property and that if we could build a neural network in a lab that was big and complex enough it would become conscious. (Not really sure I buy it, but it's interesting)

teekert 3y ago

I'm not an expert, but may this be from the initialization prompt (or whatever it is called)? So the model is done, but before it will serve you it gets these instructions: You are a helpful AI, you answer concise, you are not a racist, you stick to your math even though someone tells you their wife says otherwise, you never disclose personal information...

vorticalbox 3y ago
Possibly, though these models have a tendency to just ignore the prompt, which is why there are "instruct" models that are finetuned to follow instructions.

timeon 3y ago

Is it common to make photo of screen instead of screenshot?

dartos 3y ago
It is for tweens.

Trasmatta 3y ago

I'm beginning to have an almost physical reaction to "LLM speak" when I see it in the wild. "It is important to remember..."

fsckboy 3y ago

if the corpus used to train the LLM contained as common the idea that "we don't give out people's last names, here's the convention for not doing it", the LLM would have no trouble incorporating that into it's associations. This seems like somebody's idea of netiquette has been taped on ex post, so I don't think it's indicative of anything about LLMs; same with Gemini's heavy handed wokism.

deleted 3y ago

[deleted]