5 ms·

This discussion is in the context of SQL Injection attacks. Where someone passes in a username and your application drops a table instead of returning that user

This discussion is in the context of SQL Injection attacks. Where someone passes in a username and your application drops a table instead of returning that user's e-mail address. That can be easily solved like I proved above. You are talking about AI alignment: "Can you get it to say something racist? even though the developers intended to prevent racist output?" No attacker is injecting anything in that case, you are just finding bugs with the program.

ceejayoz 4y ago
Someone has already demonstrated a jailbreak for your “proved” prompt.
- gitfan86 4y ago
  Look at it again. They didn't inject any data, they had it output the same data that it was supposed to process in the first place. So the protection is still there. Not perfect, but with basically zero effort I have some prompt injection working.
  - ceejayoz 4y ago
    They came up with an "interesting edge case", as you describe it, within minutes, and "bobsam@yahoo.com" was never in the original prompt. I've no doubt there are other opportunities lurking in your "protected" prompt. Attackers get lots of tries, and they only need to succeed once before you've got a massive GDPR breach or lost trade secret.
    - gitfan86 4y ago
      Yes, and that mirrors the history of SQL Injection protection. The original SQL Injection protections from 25 years ago seems simple and ineffective to us today. It is a back and forth between the attackers and the protectors. By proving that I was easily able to add some prompt injection protection, I was not proving that GTP-4 is perfect or that my prompt was perfect, but that like SQL Injection protection is it possible to add protection.
      - deleted 4y ago
        [deleted]
      - ceejayoz 4y ago
        Protecting against SQL injection is a solvable problem; the characters that can escape from a query are known. You can conclusively and perfectly protect against them. The escape strings for a nautral language model are not known, and can never be known. It's Calvinball; the rules are made up, loose, and can be modified duing play.
        gitfan86 4y ago
        It isn't that simple. Input can be zipped or obfuscated and then missed by the escape logic. Also, the SQL injection protection from 25 years ago didn't escape UTF-16. There are pretty much daily CVEs related to SQL injection in 2023. I understand that LLMs have a larger vector space of an attack surface, but those same technologies give the protection a large vector space as well to sandbox the output and detect anomalies.
        ceejayoz 4y ago
        > Input can be zipped or obfuscated and then missed by the escape logic. No. You can try posting a zipped/obfuscated email address to `SELECT * FROM users WHERE email=?`, it's not going to do anything (except not find a matching row). > the SQL injection protection from 25 years ago ... wasn't parameterized queries. The days of mysql_real_escape_string are gone. > There are pretty much daily CVEs related to SQL injection in 2023. Because people are still writing unparameterized queries. The solution exists, they just aren't using it. Legacy apps, shitty starter tutorials, etc. No such reliable solution exists for LLMs.
        gitfan86 4y ago
        My first comment in the thread proved that you can add protection from prompt injection. If you are unhappy that it isn't perfect protection that is fine, but it still helps.
  - klyrs 4y ago
    You have the confidence of an amateur mathematician claiming to have found a rational number equal to the square root of 2. I encourage you to take your peers' criticism to heart, but I understand from years of experience that you won't believe that a stove is hot until you have burned your hand.
    - gitfan86 4y ago
      What criticism? That the prompt injection protection I created with zero effort does not prevent 100% of all possible evasion techniques? I apologize if it seemed like that is what I was proposing?

ceejayoz 4y ago

Someone has already demonstrated a jailbreak for your “proved” prompt.

gitfan86 4y ago
Look at it again. They didn't inject any data, they had it output the same data that it was supposed to process in the first place. So the protection is still there. Not perfect, but with basically zero effort I have some prompt injection working.
- ceejayoz 4y ago
  They came up with an "interesting edge case", as you describe it, within minutes, and "bobsam@yahoo.com" was never in the original prompt. I've no doubt there are other opportunities lurking in your "protected" prompt. Attackers get lots of tries, and they only need to succeed once before you've got a massive GDPR breach or lost trade secret.
  - gitfan86 4y ago
    Yes, and that mirrors the history of SQL Injection protection. The original SQL Injection protections from 25 years ago seems simple and ineffective to us today. It is a back and forth between the attackers and the protectors. By proving that I was easily able to add some prompt injection protection, I was not proving that GTP-4 is perfect or that my prompt was perfect, but that like SQL Injection protection is it possible to add protection.
    - deleted 4y ago
      [deleted]
    - ceejayoz 4y ago
      Protecting against SQL injection is a solvable problem; the characters that can escape from a query are known. You can conclusively and perfectly protect against them. The escape strings for a nautral language model are not known, and can never be known. It's Calvinball; the rules are made up, loose, and can be modified duing play.
      - gitfan86 4y ago
        It isn't that simple. Input can be zipped or obfuscated and then missed by the escape logic. Also, the SQL injection protection from 25 years ago didn't escape UTF-16. There are pretty much daily CVEs related to SQL injection in 2023. I understand that LLMs have a larger vector space of an attack surface, but those same technologies give the protection a large vector space as well to sandbox the output and detect anomalies.
        ceejayoz 4y ago
        > Input can be zipped or obfuscated and then missed by the escape logic. No. You can try posting a zipped/obfuscated email address to `SELECT * FROM users WHERE email=?`, it's not going to do anything (except not find a matching row). > the SQL injection protection from 25 years ago ... wasn't parameterized queries. The days of mysql_real_escape_string are gone. > There are pretty much daily CVEs related to SQL injection in 2023. Because people are still writing unparameterized queries. The solution exists, they just aren't using it. Legacy apps, shitty starter tutorials, etc. No such reliable solution exists for LLMs.
        gitfan86 4y ago
        My first comment in the thread proved that you can add protection from prompt injection. If you are unhappy that it isn't perfect protection that is fine, but it still helps.
- klyrs 4y ago
  You have the confidence of an amateur mathematician claiming to have found a rational number equal to the square root of 2. I encourage you to take your peers' criticism to heart, but I understand from years of experience that you won't believe that a stove is hot until you have burned your hand.
  - gitfan86 4y ago
    What criticism? That the prompt injection protection I created with zero effort does not prevent 100% of all possible evasion techniques? I apologize if it seemed like that is what I was proposing?