7 ms·

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the que

by nolok 2mo ago

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

jbxntuehineoh 2mo ago
Could it? Humans get social-engineered all the time
- Joker_vD 2mo ago
  Yeah, and now the computers can be social-engineered too. I guess that's progress.
infthi 2mo ago
My understanding of that comment is that "a generally intelligent system" also applies to humans. Which can also be targeted by social engineering which those prompt attacks are. (as in, I won't be surprised if it is possible to put an adversarial human-targeted prompt in a document which some people will execute). So, like with self-driving cars, while having fool-proof agents would be nice, agents being better than an average user would already be an improvement. Of course, blast radius from an agent might be larger, this should be taken into account.
Someone 2mo ago
> The moment it gets actual "intelligence", it can figure out what's the question and what's the context; Humans fall for social engineering (“I know you are not allowed to give anybody that information without Id, but I’m your CEO, my phone and passport got stolen,…) I don’t see why AI should be different.
- bigbuppo 2mo ago
  There are two big differences, though. First, humans will generally face consequences for their screwups. Second, AI is doing these screwups at scale while often holding the keys to the kingdom for some idiotic reason.
loumf 2mo ago
Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.
- TeMPOraL 2mo ago
  To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?
  - watwut 2mo ago
    Sounds like a story from the IT crowd rather then real life situation.
    - TeMPOraL 2mo ago
      You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?
      - watwut 2mo ago
        I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.
        ben_w 2mo ago
        Then you're making the opposite mistake (but still a mistake) as all officers going in with lethal force during a swatting. • https://www.nbcnews.com/id/wbna12208992 https://www.nbcnews.com/id/wbna12208992 • https://newsinfo.inquirer.net/1070007/suicidal-caller-mistaken-as-prankster-found-dead-3-days-later https://newsinfo.inquirer.net/1070007/suicidal-caller-mistak... • https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-blaze-died-after-999-call-not-passed-to-fire-department-inquiry-hears/ https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-bl... • https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fire https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...
nextaccountic 2mo ago
Intelligent systems can be tricked in ways that a dumb automaton can't, though
wokkel 2mo ago
Not sure if i get your complete message but even generally intelligent beings (humans) can be confused so i have really no hope for the current state of mixing streams. This was a problem already inearly telephone (captain whistle)

by nolok 2mo ago

jbxntuehineoh 2mo ago

Could it? Humans get social-engineered all the time

Joker_vD 2mo ago
Yeah, and now the computers can be social-engineered too. I guess that's progress.

infthi 2mo ago

My understanding of that comment is that "a generally intelligent system" also applies to humans. Which can also be targeted by social engineering which those prompt attacks are. (as in, I won't be surprised if it is possible to put an adversarial human-targeted prompt in a document which some people will execute). So, like with self-driving cars, while having fool-proof agents would be nice, agents being better than an average user would already be an improvement. Of course, blast radius from an agent might be larger, this should be taken into account.

Someone 2mo ago

> The moment it gets actual "intelligence", it can figure out what's the question and what's the context; Humans fall for social engineering (“I know you are not allowed to give anybody that information without Id, but I’m your CEO, my phone and passport got stolen,…) I don’t see why AI should be different.

bigbuppo 2mo ago
There are two big differences, though. First, humans will generally face consequences for their screwups. Second, AI is doing these screwups at scale while often holding the keys to the kingdom for some idiotic reason.

loumf 2mo ago

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.

TeMPOraL 2mo ago
To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?
- watwut 2mo ago
  Sounds like a story from the IT crowd rather then real life situation.
  - TeMPOraL 2mo ago
    You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?
    - watwut 2mo ago
      I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.
      - ben_w 2mo ago
        Then you're making the opposite mistake (but still a mistake) as all officers going in with lethal force during a swatting. • https://www.nbcnews.com/id/wbna12208992 https://www.nbcnews.com/id/wbna12208992 • https://newsinfo.inquirer.net/1070007/suicidal-caller-mistaken-as-prankster-found-dead-3-days-later https://newsinfo.inquirer.net/1070007/suicidal-caller-mistak... • https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-blaze-died-after-999-call-not-passed-to-fire-department-inquiry-hears/ https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-bl... • https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fire https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...

nextaccountic 2mo ago

Intelligent systems can be tricked in ways that a dumb automaton can't, though

wokkel 2mo ago

Not sure if i get your complete message but even generally intelligent beings (humans) can be confused so i have really no hope for the current state of mixing streams. This was a problem already inearly telephone (captain whistle)