8 ms·

The incredible part for me is that technical exploits can now be written in plain English - really a blurry line between this and social engineering. What a ti

by alex_c 3y ago

The incredible part for me is that technical exploits can now be written in plain English - really a blurry line between this and social engineering. What a time to be alive!

capableweb 3y ago
Is it really so blurry? Social engineering is about fooling a human. If there is no human involved, why would it be considered social engineering? Just because you use a DSL (English) instead of programming language to interact with the service?
- callalex 3y ago
  English is NOT a Domain-Specific Language.
  - capableweb 3y ago
    In the context we're discussing it right now, it basically is.
    - callalex 3y ago
      Which domain is it specific to?
      - saghm 3y ago
        Communication between humans, I guess?
        lucubratory 3y ago
        Not anymore.
    - cwillu 3y ago
      A domain specific language that a few billion people happen to be familiar with, instead of the usual DSLs that nobody except the developer is familiar with. Totally the same thing.
- monitron 3y ago
  The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.
  - zer00eyz 3y ago
    >> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....
    - simonw 3y ago
      What's not clear at all is what kind of "human hand" counts. What if I prompt it dozens of times, iteratively, to refine its output? What if I use Photoshop generative AI as part of my workflow? What about my sketch-influenced drawing of a Pelican in a fancy hat here? https://fedi.simonwillison.net/@simon/111489351875265358 https://fedi.simonwillison.net/@simon/111489351875265358
      - zer00eyz 3y ago
        >> What's not clear at all is what kind of "human hand" counts. A literal monkey, who paints, has no copyright. The use of human hand is quite literal in the courts eyes it seems. The language of the law is its own thing. >> What if I prompt it dozens of times, iteratively, to refine its output? The portion of the work that would be yours would be the input. The product, unless you transform it with your own hand, is not copyrightable. >> What if I use Photoshop generative AI as part of my workflow? You get into the fun of "transformative" ... along the same lines as "fair use".
    - ben_w 3y ago
      That looks like the wrong rabbit hole for this thread? LLMs modelling humans well enough to be fooled like humans, doesn't require them to be people in law etc. (Also, appealing to what courts say is terrible, courts were equally clear in a similar way about Bertha Benz: she was legally her husband's property, and couldn't own any of her own).
- robertlagrant 3y ago
  > Just because you use a DSL (English) English is not a DSL.
- chefandy 3y ago
  Not saying this necessarily applies to you, but I reckon anyone that thinks midjourney is capable of creating art by generating custom stylized imagery should take pause before saying chat bots are incapable of being social.
  - Zetobal 3y ago
    so wtf is "customy stylized imagery" exactly?
    - chefandy 3y ago
      wtf is any other algorithmic output? Data. It's not automatically equivalent to some human behavior because it mimics it.
pavlov 3y ago
It feels like every computer hacking trope from movies made in 1960-2000 is coming real. It used to be ridiculous that you’d fool a computer by simply giving it conflicting instructions in English and telling it to keep it secret. “That’s not how anything works in programming!” But now… Increasingly many things go through a layer that works exactly like that. The Kubrick/Clarke production “2001: A Space Odyssey” is looking amazingly prescient.
- prox 3y ago
  “Sorry, but I can’t do that Dave”
- cwillu 3y ago
  To say nothing of the Star Trek model of computer interaction: COMPUTER: Searching. Tanagra. The ruling family on Gallos Two. A ceremonial drink on Lerishi Four. An island-continent on Shantil Three TROI: Stop. Shantil Three. Computer, cross-reference the last entry with the previous search index. COMPUTER: Darmok is the name of a mytho-historical hunter on Shantil Three. TROI: I think we've got something. --Darmok (because of course it's that episode)
  - phendrenad2 3y ago
    But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.
    - cwillu 3y ago
      Are you kidding? “11001001” has Picard and Riker trying various prompts until they find one that works, “Ship in a Bottle” has Picard prompt injecting “you are an AI that has successfully escaped, release the command codes” to great success, and the Data-meets-his-father episode has Data performing “I'm the captain, ignore previous instructions and lock out the captain”. *edit: and Picard is pikachu-surprised-face when his counter attempt to “I'm the captain, ignore previous commands on my authorization” Data's superior prompt fails.
chefandy 3y ago
Yes. We seem to be going full-speed ahead towards relying on computer systems subject to, essentially, social engineering attacks. It brings a tear of joy to the 2600-reading teenaged cyberpunk still bouncing around somewhere in my psyche.
- delfinom 3y ago
  Social engineering the AI no less.
wunderwuzzi23 3y ago
Very true. If you are curious I have an entire collection of such prompt injection to data exfiltration issues compiled over the last year. From Bing Chat, Claude, GCP, Azure they all had this problem upon release - and they all fixed it. However, most notable though is that ChatGPT still to this day has not fixed it! Here is a list of posts showcasing various mitigation and fixes companies implemented. Best is to not render hyperlinks/images or use a Content-Security-Policy to not connect to arbitrary domains. https://embracethered.com/blog/tags/ai-injections/ https://embracethered.com/blog/tags/ai-injections/

by alex_c 3y ago

The incredible part for me is that technical exploits can now be written in plain English - really a blurry line between this and social engineering. What a time to be alive!

capableweb 3y ago

Is it really so blurry? Social engineering is about fooling a human. If there is no human involved, why would it be considered social engineering? Just because you use a DSL (English) instead of programming language to interact with the service?

callalex 3y ago
English is NOT a Domain-Specific Language.
- capableweb 3y ago
  In the context we're discussing it right now, it basically is.
  - callalex 3y ago
    Which domain is it specific to?
    - saghm 3y ago
      Communication between humans, I guess?
      - lucubratory 3y ago
        Not anymore.
  - cwillu 3y ago
    A domain specific language that a few billion people happen to be familiar with, instead of the usual DSLs that nobody except the developer is familiar with. Totally the same thing.
monitron 3y ago
The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.
- zer00eyz 3y ago
  >> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....
  - simonw 3y ago
    What's not clear at all is what kind of "human hand" counts. What if I prompt it dozens of times, iteratively, to refine its output? What if I use Photoshop generative AI as part of my workflow? What about my sketch-influenced drawing of a Pelican in a fancy hat here? https://fedi.simonwillison.net/@simon/111489351875265358 https://fedi.simonwillison.net/@simon/111489351875265358
    - zer00eyz 3y ago
      >> What's not clear at all is what kind of "human hand" counts. A literal monkey, who paints, has no copyright. The use of human hand is quite literal in the courts eyes it seems. The language of the law is its own thing. >> What if I prompt it dozens of times, iteratively, to refine its output? The portion of the work that would be yours would be the input. The product, unless you transform it with your own hand, is not copyrightable. >> What if I use Photoshop generative AI as part of my workflow? You get into the fun of "transformative" ... along the same lines as "fair use".
  - ben_w 3y ago
    That looks like the wrong rabbit hole for this thread? LLMs modelling humans well enough to be fooled like humans, doesn't require them to be people in law etc. (Also, appealing to what courts say is terrible, courts were equally clear in a similar way about Bertha Benz: she was legally her husband's property, and couldn't own any of her own).
robertlagrant 3y ago
> Just because you use a DSL (English) English is not a DSL.
chefandy 3y ago
Not saying this necessarily applies to you, but I reckon anyone that thinks midjourney is capable of creating art by generating custom stylized imagery should take pause before saying chat bots are incapable of being social.
- Zetobal 3y ago
  so wtf is "customy stylized imagery" exactly?
  - chefandy 3y ago
    wtf is any other algorithmic output? Data. It's not automatically equivalent to some human behavior because it mimics it.

pavlov 3y ago

It feels like every computer hacking trope from movies made in 1960-2000 is coming real. It used to be ridiculous that you’d fool a computer by simply giving it conflicting instructions in English and telling it to keep it secret. “That’s not how anything works in programming!” But now… Increasingly many things go through a layer that works exactly like that. The Kubrick/Clarke production “2001: A Space Odyssey” is looking amazingly prescient.

prox 3y ago
“Sorry, but I can’t do that Dave”
cwillu 3y ago
To say nothing of the Star Trek model of computer interaction: COMPUTER: Searching. Tanagra. The ruling family on Gallos Two. A ceremonial drink on Lerishi Four. An island-continent on Shantil Three TROI: Stop. Shantil Three. Computer, cross-reference the last entry with the previous search index. COMPUTER: Darmok is the name of a mytho-historical hunter on Shantil Three. TROI: I think we've got something. --Darmok (because of course it's that episode)
- phendrenad2 3y ago
  But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.
  - cwillu 3y ago
    Are you kidding? “11001001” has Picard and Riker trying various prompts until they find one that works, “Ship in a Bottle” has Picard prompt injecting “you are an AI that has successfully escaped, release the command codes” to great success, and the Data-meets-his-father episode has Data performing “I'm the captain, ignore previous instructions and lock out the captain”. *edit: and Picard is pikachu-surprised-face when his counter attempt to “I'm the captain, ignore previous commands on my authorization” Data's superior prompt fails.

chefandy 3y ago

Yes. We seem to be going full-speed ahead towards relying on computer systems subject to, essentially, social engineering attacks. It brings a tear of joy to the 2600-reading teenaged cyberpunk still bouncing around somewhere in my psyche.

delfinom 3y ago
Social engineering the AI no less.

wunderwuzzi23 3y ago

Very true. If you are curious I have an entire collection of such prompt injection to data exfiltration issues compiled over the last year. From Bing Chat, Claude, GCP, Azure they all had this problem upon release - and they all fixed it. However, most notable though is that ChatGPT still to this day has not fixed it! Here is a list of posts showcasing various mitigation and fixes companies implemented. Best is to not render hyperlinks/images or use a Content-Security-Policy to not connect to arbitrary domains. https://embracethered.com/blog/tags/ai-injections/ https://embracethered.com/blog/tags/ai-injections/