3 ms·

I fail to see how these are attacks and not just the model giving you what conforms to the original prompt text the most according to a seed value.

navjack27 4y ago
It's not alive. it's not thinking. It has access to nothing. It's just making text. This isn't magic. It's not like it is some sort of thing that has admin access to the server and computer it's executed in. It's code. It's data.
- yunwal 4y ago
  The point is that it's unsafe to do anything critical with the output of GPT-3 without running it by a human. It's not going to replace human cognition in safety-critical systems unless there's a solution to this.
jannes 4y ago
Do you consider social engineering to be an attack?
- navjack27 4y ago
  If the social engineering actually results in the person being engineered giving away actual secrets then yes. If the person being engineered simply lies to you with something that appears to be the secrets then no.
Frost1x 4y ago
I don't know how this is executed in reality but if the GPT instances aren't well isolated, it seems like you may be able to get the model to execute system level code or leak prompts and information from others using the same instance. Again, I don't know how well isolated these instances are in practice. Outside of that I tend to agree with you, you're simply poking at your own isolated instance and getting back what you expect, so who cares?
snowwrestler 4y ago
This is like saying SQL injection is not an attack because the database engine is just giving you the correct response for the query it received. Computing systems do what they are told, yes. What makes a behavior an “attack” is whether it intentionally inhibits or harms the purpose to which the owner has set the computing system. In this case, yes, these are just interesting prompts and responses. But in the case of a system X running on top of the model Y and engineered to expect certain responses from Y, subverting the expected behavior of X purely by varying user input to Y could be perceived as harm to X. Just like a SQL injection harms the application without exploiting the DB engine.
- xani_ 4y ago
  Well, exactly that, it's not attack on a database, because SQL does provide way to pass inputs correctly, just on the code that connects to database and allows that. I.e. the fault is at amateurish code before the SQL/GPT, not the "database" itself

navjack27 4y ago

It's not alive. it's not thinking. It has access to nothing. It's just making text. This isn't magic. It's not like it is some sort of thing that has admin access to the server and computer it's executed in. It's code. It's data.

yunwal 4y ago
The point is that it's unsafe to do anything critical with the output of GPT-3 without running it by a human. It's not going to replace human cognition in safety-critical systems unless there's a solution to this.

jannes 4y ago

Do you consider social engineering to be an attack?

navjack27 4y ago
If the social engineering actually results in the person being engineered giving away actual secrets then yes. If the person being engineered simply lies to you with something that appears to be the secrets then no.

Frost1x 4y ago

I don't know how this is executed in reality but if the GPT instances aren't well isolated, it seems like you may be able to get the model to execute system level code or leak prompts and information from others using the same instance. Again, I don't know how well isolated these instances are in practice. Outside of that I tend to agree with you, you're simply poking at your own isolated instance and getting back what you expect, so who cares?

snowwrestler 4y ago

This is like saying SQL injection is not an attack because the database engine is just giving you the correct response for the query it received. Computing systems do what they are told, yes. What makes a behavior an “attack” is whether it intentionally inhibits or harms the purpose to which the owner has set the computing system. In this case, yes, these are just interesting prompts and responses. But in the case of a system X running on top of the model Y and engineered to expect certain responses from Y, subverting the expected behavior of X purely by varying user input to Y could be perceived as harm to X. Just like a SQL injection harms the application without exploiting the DB engine.

xani_ 4y ago
Well, exactly that, it's not attack on a database, because SQL does provide way to pass inputs correctly, just on the code that connects to database and allows that. I.e. the fault is at amateurish code before the SQL/GPT, not the "database" itself