4 ms·

I’d say it’s not the same thing, because most humans don’t have an encyclopedic knowledge of past scams, and are not primed to watch out for them 24/7. LLMs don

I’d say it’s not the same thing, because most humans don’t have an encyclopedic knowledge of past scams, and are not primed to watch out for them 24/7. LLMs don’t have either of these problems. An interesting question is whether GPT-4 would fall for a phishing scam or try to buy a timeshare if you gave it an explicit instruction to avoid being scammed.

danpalmer 3y ago
I sort of disagree that LLMs don’t have the same pitfalls. LLMs aren’t recording everything they are trained with, like humans, the training data affects a general behavioural model. When answering, they aren’t looking up information. As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam. Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work. To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.
amelius 3y ago
The problem is that the attacker can try a gazillion times and only needs to succeed once. This is where it is different from the human case, where the human will get bored after 3 phishing attempts and closes their email program.

danpalmer 3y ago

I sort of disagree that LLMs don’t have the same pitfalls. LLMs aren’t recording everything they are trained with, like humans, the training data affects a general behavioural model. When answering, they aren’t looking up information. As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam. Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work. To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.

amelius 3y ago

The problem is that the attacker can try a gazillion times and only needs to succeed once. This is where it is different from the human case, where the human will get bored after 3 phishing attempts and closes their email program.