5 ms·

I'm concerned that OpenAI's example documentation suggests using this to A) construct SQL queries and B) summarize emails, but that their example code doesn't i

I'm concerned that OpenAI's example documentation suggests using this to A) construct SQL queries and B) summarize emails, but that their example code doesn't include clear hooks for human validation before actions are called. For a recipe builder it's not so big a deal, but I really worry how eager people are to remove human review from these steps. It gets rid of a very important mechanism for reducing the risks of prompt injection. The top comment here suggests wiring this up to allow GPT-4 to recursively call itself. Meanwhile, some of the best advice I've seen from security professionals on secure LLM app development is to whenever possible completely isolate queries from each other to reduce the potential damage that a compromised agent can do before its "memory" is wiped. There are definitely ways to use this safely, and there are definitely some pretty powerful apps you could build on top of this without much risk. LLMs as a transformation layer for trusted input is a good use-case. But are devs going to stick with that? Is it going to be used safely? Do devs understand any of the risks or how to mitigate them in the first place? 3rd-party plugins on ChatGPT have repeatedly been vulnerable in the real world, I'm worried about what mistakes developers are going to make now that they're actively encouraged to treat GPT as even more of a low-level data layer. Especially since OpenAI's documentation on how to build secure apps is mostly pretty bad, and they don't seem to be spending much time or effort educating developers/partners on how to approach LLM security.

abhibeckert 3y ago
In my opinion the only way to use it safely is to ensure your AI only has access to data that the end user already has access to. At that point, prompt injection is no-longer an issue - because the AI doesn't need to hide anything. Giving GPT access to your entire database, but telling it not to reveal certain bits, is never going to work. There will always be side channel vulnerabilities in those systems.
- jacobr1 3y ago
  > your AI only has access to data that the end user already has access to. That doesn't work for the same reason you mention with a DB ... any data source is vulnerable to indirect injection attacks. If you open the door to ANY data source this a factor, including ones under the sole "control" of the user.
- danShumway 3y ago
  > e.g. define a function called extract_data(name: string, birthday: string), or sql_query(query: string) This section in OpenAI's product announcement really irritates me because it's so obvious that the model should have access to a subset of API calls that themselves fetch the data, as opposed to giving the model raw access to SQL. You could have the same capabilities while eliminating a huge amount of risk. And OpenAI just sticks this right in the announcement, they're encouraging it. When I'm building a completely isolated backend with just regular code, I still usually put a data access layer in front of the database in most cases. I still don't want my REST endpoints directly building SQL queries or directly accessing the database, and that's without an LLM in the loop at all. It's just safer. It's the same idea as using `innerHTML`; in general it's better when possible to have those kinds of calls extremely isolated and to go through functions that constrain what can go wrong. But no, OpenAI just straight up telling developers to do the wrong things and to give GPT unrestricted database access.
  - BoorishBears 3y ago
    You don't need to directly run the query it returns, you can use that query as a sub-query on a known safe set of data and let it fail if someone manages to prompt inject their way into looking at other tables/columns. That way you can support natural language to query without sending dozens of functions (which will eat up the context window)
    - danShumway 3y ago
      You can do that (I wouldn't advise it, there are still problems that are better solved by building explicit functions; but you can use subqueries and it would be safer) -- but most developers won't. They'll run the query directly. Most developers also will not execute it as a readonly query, they'll give the LLM write access to the database. If OpenAI doesn't know that, then I don't know what to say, they haven't spent enough time writing documentation for general users.
      - BoorishBears 3y ago
        You can't advise for or against it without a well defined problem: for some cases explicit functions won't even be an option. Defining basic CRUD functions for a few basic entities will a ton of tokens in schema definitions, and still suffers from injection if you want to support querying on data that wasn't well defined a-priori, which is a problem I've worked on. Overall if this was one of their example projects I'd be disappointed, but it was a snippet in a release note. So far their actual example projects have done a fair job showing where guardrails in production systems are needed, I wouldn't over-index on this.
- kristiandupont 3y ago
  >At that point, prompt injection is no-longer an issue [...] As far as input goes, yes. But I am more worried about agents that can take actions that affect the outside world, like sending emails on your behalf.
sillysaurusx 3y ago
I was going to say “I look forward to it and think it’s hilarious,” but then I remembered that most victims will be people learning to code, not companies. It would really suck to suddenly lose your recipe database when you just wanted to figure out how this programming stuff worked. Some kind of “heads up” tagline is probably a good idea, yeah.
- kristiandupont 3y ago
  I think the victims will mostly be the users of the software. The personal assistant that can handle your calendar and emails and all would be able to do real damage.
irthomasthomas 3y ago
I don't understand why they have done this? Like, how did the conversations go when it was pointed out to them what a pretty darn bad idea it was to recommend connecting chatgpt directly to a SQL database? I know we are supposed to assume incompetence over malice, but no one is that incompetent. They must have had the conversations, and chose to do it anyway.
- sebzim4500 3y ago
  Why is this unreasonable to you? I can imagine using this, just run it with read access and check the sql if the results are interesting.
  - irthomasthomas 3y ago
    Even read only. You are giving access to your data to a black box API.
    - sebzim4500 3y ago
      If it's on Azure anyway I don't see the big deal, especially if you are an enterprise and so buying it via azure instead of directly.
- blitzar 3y ago
  Perhaps they plan on having ChatGPT make a quick copy of your database, for your convenience of course.

abhibeckert 3y ago

In my opinion the only way to use it safely is to ensure your AI only has access to data that the end user already has access to. At that point, prompt injection is no-longer an issue - because the AI doesn't need to hide anything. Giving GPT access to your entire database, but telling it not to reveal certain bits, is never going to work. There will always be side channel vulnerabilities in those systems.

jacobr1 3y ago
> your AI only has access to data that the end user already has access to. That doesn't work for the same reason you mention with a DB ... any data source is vulnerable to indirect injection attacks. If you open the door to ANY data source this a factor, including ones under the sole "control" of the user.
danShumway 3y ago
> e.g. define a function called extract_data(name: string, birthday: string), or sql_query(query: string) This section in OpenAI's product announcement really irritates me because it's so obvious that the model should have access to a subset of API calls that themselves fetch the data, as opposed to giving the model raw access to SQL. You could have the same capabilities while eliminating a huge amount of risk. And OpenAI just sticks this right in the announcement, they're encouraging it. When I'm building a completely isolated backend with just regular code, I still usually put a data access layer in front of the database in most cases. I still don't want my REST endpoints directly building SQL queries or directly accessing the database, and that's without an LLM in the loop at all. It's just safer. It's the same idea as using `innerHTML`; in general it's better when possible to have those kinds of calls extremely isolated and to go through functions that constrain what can go wrong. But no, OpenAI just straight up telling developers to do the wrong things and to give GPT unrestricted database access.
- BoorishBears 3y ago
  You don't need to directly run the query it returns, you can use that query as a sub-query on a known safe set of data and let it fail if someone manages to prompt inject their way into looking at other tables/columns. That way you can support natural language to query without sending dozens of functions (which will eat up the context window)
  - danShumway 3y ago
    You can do that (I wouldn't advise it, there are still problems that are better solved by building explicit functions; but you can use subqueries and it would be safer) -- but most developers won't. They'll run the query directly. Most developers also will not execute it as a readonly query, they'll give the LLM write access to the database. If OpenAI doesn't know that, then I don't know what to say, they haven't spent enough time writing documentation for general users.
    - BoorishBears 3y ago
      You can't advise for or against it without a well defined problem: for some cases explicit functions won't even be an option. Defining basic CRUD functions for a few basic entities will a ton of tokens in schema definitions, and still suffers from injection if you want to support querying on data that wasn't well defined a-priori, which is a problem I've worked on. Overall if this was one of their example projects I'd be disappointed, but it was a snippet in a release note. So far their actual example projects have done a fair job showing where guardrails in production systems are needed, I wouldn't over-index on this.
kristiandupont 3y ago
>At that point, prompt injection is no-longer an issue [...] As far as input goes, yes. But I am more worried about agents that can take actions that affect the outside world, like sending emails on your behalf.

sillysaurusx 3y ago

I was going to say “I look forward to it and think it’s hilarious,” but then I remembered that most victims will be people learning to code, not companies. It would really suck to suddenly lose your recipe database when you just wanted to figure out how this programming stuff worked. Some kind of “heads up” tagline is probably a good idea, yeah.

kristiandupont 3y ago
I think the victims will mostly be the users of the software. The personal assistant that can handle your calendar and emails and all would be able to do real damage.

irthomasthomas 3y ago

I don't understand why they have done this? Like, how did the conversations go when it was pointed out to them what a pretty darn bad idea it was to recommend connecting chatgpt directly to a SQL database? I know we are supposed to assume incompetence over malice, but no one is that incompetent. They must have had the conversations, and chose to do it anyway.

sebzim4500 3y ago
Why is this unreasonable to you? I can imagine using this, just run it with read access and check the sql if the results are interesting.
- irthomasthomas 3y ago
  Even read only. You are giving access to your data to a black box API.
  - sebzim4500 3y ago
    If it's on Azure anyway I don't see the big deal, especially if you are an enterprise and so buying it via azure instead of directly.
blitzar 3y ago
Perhaps they plan on having ChatGPT make a quick copy of your database, for your convenience of course.