5 ms·

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. A

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing. I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults. LLMs show childlike intelligence in this regard while being more adult in others.

draw_down 3y ago
[dead]
ckrapu 3y ago
I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?
- hxugufjfjf 3y ago
  Yes there is. The paper describing GPT4 had a section on this.
- amelius 3y ago
  How about: unreliable child savant. You can use it in sentences such as: Would you let your children talk to the unreliable child savant?
pyth0 3y ago
The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.
- thornewolf 3y ago
  "It merely has all of the byproducts of intelligence, its not intelligence though!" I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be measured in the same way a humans intelligence does. What I do claim though is that it is intelligence - intelligence that relates to humans in the same way the mind of a crow might. The dominant mindset I have observed in my life thus-far when people discuss human intelligence is the pattern matching perspective. Humans are differentiates by our outsized ability to pattern match being able to successfully manipulate these patterns. We now see something nonorganic with amazing pattern matching abilities. We have previously seen other organic entities with impressive pattern matching abilities. Why must this situation be any different? My overall claims: - Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence (however that existence may manifest) - Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence" - Considering something intelligent or to have desires is not anthropomorphizing. There are many non-human entities that we consider to have these properties.
  - pyth0 3y ago
    > Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence. I agree, although I think we humans have always been fairly bad at measuring intelligence in a way that truly appreciates all the complexity of it. The second part of that is also interesting and I would agree that is partly what makes these LLMs non-intelligent. The models do not really have "an existence" outside of the moment in which they are processing the context and producing output. > Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence" I don't think I agree with this, or at least maybe I disagree with your definition of "intelligent". I believe that intelligence is heavily intertwined with biology and it exists is all manner of non-human creatures but I don't think I would call an entirely mechanical system "intelligent". Perhaps I would say it had "intelligent design". > Considering something intelligent or to have desires is not anthropomorphizing. I absolutely agree with this and I was not trying to imply that it was unique to humans. In fact I think we severely discount the amount of intelligence in non-human life forms all the time. I do think that ChatGPT possesses knowledge (as encoded in its weights) similar to a book, however unlike a book it also has a convenient and familiar interface that allows us to interact with this knowledge and form unique and novel results.

draw_down 3y ago

[dead]

ckrapu 3y ago

I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?

hxugufjfjf 3y ago
Yes there is. The paper describing GPT4 had a section on this.
amelius 3y ago
How about: unreliable child savant. You can use it in sentences such as: Would you let your children talk to the unreliable child savant?

pyth0 3y ago

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

thornewolf 3y ago
"It merely has all of the byproducts of intelligence, its not intelligence though!" I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be measured in the same way a humans intelligence does. What I do claim though is that it is intelligence - intelligence that relates to humans in the same way the mind of a crow might. The dominant mindset I have observed in my life thus-far when people discuss human intelligence is the pattern matching perspective. Humans are differentiates by our outsized ability to pattern match being able to successfully manipulate these patterns. We now see something nonorganic with amazing pattern matching abilities. We have previously seen other organic entities with impressive pattern matching abilities. Why must this situation be any different? My overall claims: - Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence (however that existence may manifest) - Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence" - Considering something intelligent or to have desires is not anthropomorphizing. There are many non-human entities that we consider to have these properties.
- pyth0 3y ago
  > Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence. I agree, although I think we humans have always been fairly bad at measuring intelligence in a way that truly appreciates all the complexity of it. The second part of that is also interesting and I would agree that is partly what makes these LLMs non-intelligent. The models do not really have "an existence" outside of the moment in which they are processing the context and producing output. > Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence" I don't think I agree with this, or at least maybe I disagree with your definition of "intelligent". I believe that intelligence is heavily intertwined with biology and it exists is all manner of non-human creatures but I don't think I would call an entirely mechanical system "intelligent". Perhaps I would say it had "intelligent design". > Considering something intelligent or to have desires is not anthropomorphizing. I absolutely agree with this and I was not trying to imply that it was unique to humans. In fact I think we severely discount the amount of intelligence in non-human life forms all the time. I do think that ChatGPT possesses knowledge (as encoded in its weights) similar to a book, however unlike a book it also has a convenient and familiar interface that allows us to interact with this knowledge and form unique and novel results.