7 ms·

> It is possible to use these unreleased plugins by setting up match-and-replace rules through an HTTP proxy. There are only client-side checks to validate that

by koolba 4y ago

> It is possible to use these unreleased plugins by setting up match-and-replace rules through an HTTP proxy. There are only client-side checks to validate that you have permission to use the plugins and they can be bypassed. There’s no way I’m going to accept the intersection of “we take security very seriously” and implementing security checks purely client side. This and the recent title information leak are both canaries for how the rest of Open AI operates.

sebzim4500 4y ago
Maybe they don't consider it a security check.
- LordDragonfang 4y ago
  It's possible that someone on the security team knew this would happen and wanted the competent pentesters to find those plugins and give them more real-world red-team data. I only bring this up because that was the whole purpose of publicly releasing chatgpt in the first place, so it's known to be part of their strategy.
  - techdragon 4y ago
    The entire attitude to how they are managing use of ChatGPT in general does feel a lot like “we’re just going to treat the public as the red team. I honestly expected to see way more “I just tried a jailbreak prompt for $stupid_reason and got banned…” stories. I expected more automated front end countermeasures… to jailbreak prompt engineering efforts… but instead they appear to have nothing, not even some heuristics kicking in on sensitive wordsz
    - sebzim4500 4y ago
      Why would they ban you for trying jailbreak prompts? Wouldn't that defeat the whole purpose?
      - Filligree 4y ago
        It does break the ToS, and for puritan rules like OpenAI's it's common to expect them to be offended by anyone working around the tooling.
        textninja 4y ago
        I’m aghast that using a general purpose AI for general purposes is treated as a malicious act and censured as such. “No! Not those tokens!!”
      - techdragon 4y ago
        That’s my point. The jailbreaks are if not explicitly against the terms of service, they are certainly implicitly against the terms of service and within OpenAI’s discretion to ban people for. Yet they appear to be letting people get away with a lot. I mean heck just the other day based on a HN comment on another topic, I whipped up an Jailbreak prompt to see how hard it is to get ChatGPT to flirt or be sexual, and how well it does at writing that kind of thing. Now sure it was just a one off experiment and while they were dealing with the history incident, but it’s not like it’s hard to have something flag the three letters “sex” and have some kind of telemetry, because they explicitly mention sexual content in the TOS… they aren’t targeting the low hanging fruit because it’s pointless. They’re letting the red team (the public) try and really test the boundaries to make sure that much more serious issues aren’t hiding, waiting for someone to find much later if they were more aggressively limiting the amount of implicit TOS violations that appear to be happening.
  - sebzim4500 4y ago
    I think it's far more likely they just didn't care very much if people play with unreleased plugins before they were supposed to.
  - thricegr8 4y ago
    Can you expound on this?
CGamesPlay 4y ago
I interpreted this to mean that the plugins are actually client-side software, so of course there are client-side checks to confirm the allowed usage. That's why I can use a client-side proxy to access them. Right?
- jeremyjh 4y ago
  No, the plugins are server-side software, but the server isn't checking if the user is allowed to use the plugin requested by the client.
gjsman-1000 4y ago
I think this is a perfect example about how you could be extremely smart in one field, and extremely dumb in another. You can have a programmer brilliant at AI who doesn't know the slightest thing about web app security.
- rjzzleep 4y ago
  Most don't. But many don't think that they should reinvent web security and that is the key difference here.
- kodah 4y ago
  This also translates on to Software Engineers when talking about Systems Engineering topics. People are quite silo'd and there's often people who have a specific overlap between two disciplines.
- precompute 4y ago
  I think instead of making excuses about this, it's rational to expect a billion dollar "non-profit" to hire a team of competent web devs / backend people for their product that is only accessible via the internet.
efficax 4y ago
maybe they asked chatgpt if it was secure and trusted the response
Cthulhu_ 4y ago
It feels like the people at OpenAI have been pushed hard to get these features out the door - with the release of ChatGPT, suddenly many big players have rushed to release their (GPT 3/4) based products, which will involve multi-billion contracts. If they had waited for things like security reviews, they might have missed the boat. Developments are going really fast in this space.
- fisf 4y ago
  If things like the issues we have seen in the last days require a security review in order to be caught, then I am not very optimistic about the rest of their stack.
  - kingkawn 4y ago
    Don’t worry, we’ll have the AI secure it’s own code, it’ll go fiiiiine
- pixl97 4y ago
  "We were going to implement AI safety, but like the competitors are going to fast and we needed to release first" We are going to great filter ourselves.
  - jacquesm 4y ago
    There is a fantastic bit in one of Terry Pratchett's books: "Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."
    - pixl97 4y ago
      Oh dear, that may be me. I was at Home Depot in the checkout line and they had some construction going on. Hanging down from the ceiling by wires was a red button. I was compelled, I could not avoid it. I pushed the red button. A loud buzzing noise occurred. The cashier sighed and said that I was the sixth guy to do that, not a single woman had pressed the button so far.
axg11 4y ago
I don't think _this_ is a security check. These are essentially unlisted plugins. They will be released/approved and this is a way to allow some user to test the plugins (different client side software).
- ethanbond 4y ago
  Have they been approved and released or not? If no, then they’re not meant to be accessible, meaning it’s a security flaw to be able to access them.
  - IanCal 4y ago
    What exactly is the security issue? The thing that's affected here is the responses you alone see.
    - djbusby 4y ago
      They are saying that client-side only "ACL" is sloppy and that could be an indication of even more internal slop (of which title-leak may be another symptom)
      - londons_explore 4y ago
        I suspect it was a deliberate decision not to ACL plugins. They let anyone create and use one after all. The only reason approval exists at all is so users aren't tricked into running low quality or spammy plugins. You could consider this similar to someone revealing that lots of apps banned from app stores are available on other websites and one could write the headline "banned app leaks onto apkmirror.com, is google security compromised?"
        ethanbond 4y ago
        No, you could consider this similar to someone revealing that lots of apps banned from app stores are available on the same app stores that they’re banned from, which, yeah, looks a bit dodgy.
jnwatson 4y ago
If you on-purpose bypass a filter to use an unapproved plugin, it is on the user at that point.
lavezzi 4y ago
Funny you say that because OpenAI just sent out an email saying "another user may have inadvertently seen your billing information when clicking on their own “Manage Subscription” page." for a nine hour window yesterday
- cma 4y ago
  They didn't have the billing system at launch, so maybe it was coded with the help of chatGPT.
- jwestbury 4y ago
  Ah, yes, the Valve-patented "let's cache user account pages in the CDN" strategy.
  - _piif 4y ago
    Did they also have a similar incident before?
    - EthicalSimilar 4y ago
      Yes, a few years ago. It is the first thing I thought of when I saw this news. For a brief period of time (a few hours?), Steam would show you as being logged in as another user. You could view their billing page and other restricted pages.
yakz 4y ago
When I rewrite the features response and it then queries the plug-in list, the server responds with a 401. So maybe not purely client side (I have a plus account but no plug-in access yet).
magic_hamster 4y ago
If plugins are just text prefixes, which reside on the server and have been approved, then how is it more risky than accepting prompts in general?
salawat 4y ago
It isn't about security, it's about controlling the default baseline ecosystem. If it were truly about security A) it'd actually be secure B) it'd be way more user-hostile than a client-side ACL.
moneywoes 4y ago
Title information leak?
danShumway 4y ago
My conspiracy theory is that all of the "we take security very seriously" talk out of OpenAI has aligned with "our AI is so advanced and powerful that we have to take these existential risks very seriously and we have to make sure it doesn't turn evil." And OpenAI doesn't seem very interested in security conversations that don't feed into that narrative[0]. I think it's (mostly) just propaganda. OpenAI focuses on dangers that make their AI seem more advanced than it is, and ignore dangers that don't play well to the press (like clientside validation). I'm not even sure it's standard "we care about your security" company talk, I think it's mostly designed just to make people think of GPT as more cutting edge. The "GPT paid someone to solve a captcha" release seems like pure publicity stunt to me, given that solving captchas by pretending to be blind is something other systems can already do without human help. But it played well in press releases. ---- [0]: To be fair, they have been focusing on alignment, but I honestly don't think alignment is a security measure, I think it's a general performance measure. We haven't seen strong evidence that alignment can stop prompt injection attacks, so I think mostly OpenAI mostly just cares about the potential negative press from their AI randomly insulting people.
- precompute 4y ago
  OpenAI is 50% product, 50% advertising.
  - tmpz22 4y ago
    * 50% product, 50% fundraising
    - naniwaduni 4y ago
      Compromise: 50% advertising, 50% fundraising?
  - Balgair 4y ago
    I mean, that's a lot better than most businesses.
    - 4y ago
adambard 4y ago
So it turns out that the way ChatGPT plugins operate, any user can use unverified plugins from arbitrary urls, just by pasting in a URL: https://platform.openai.com/docs/plugins/introduction/plugin-flow https://platform.openai.com/docs/plugins/introduction/plugin... With this in mind, how concerned should we really be about a pre-registered list of unverified plugins? The only new information exposed is the URLs of these various plugins, is this information so private if it was already registered in their list to begin with?
antihero 4y ago
Perhaps the AI guys are insane as the Bitcoin guys
Palpatineli 4y ago
What can you expect from a company releasing a (at least) village idiot level AI and give it access to the internet AND a shell/Python interpreter with skin-deep alignment that is effortlessly circumvented with DAN?

by koolba 4y ago

sebzim4500 4y ago

Maybe they don't consider it a security check.

LordDragonfang 4y ago
It's possible that someone on the security team knew this would happen and wanted the competent pentesters to find those plugins and give them more real-world red-team data. I only bring this up because that was the whole purpose of publicly releasing chatgpt in the first place, so it's known to be part of their strategy.
- techdragon 4y ago
  The entire attitude to how they are managing use of ChatGPT in general does feel a lot like “we’re just going to treat the public as the red team. I honestly expected to see way more “I just tried a jailbreak prompt for $stupid_reason and got banned…” stories. I expected more automated front end countermeasures… to jailbreak prompt engineering efforts… but instead they appear to have nothing, not even some heuristics kicking in on sensitive wordsz
  - sebzim4500 4y ago
    Why would they ban you for trying jailbreak prompts? Wouldn't that defeat the whole purpose?
    - Filligree 4y ago
      It does break the ToS, and for puritan rules like OpenAI's it's common to expect them to be offended by anyone working around the tooling.
      - textninja 4y ago
        I’m aghast that using a general purpose AI for general purposes is treated as a malicious act and censured as such. “No! Not those tokens!!”
    - techdragon 4y ago
      That’s my point. The jailbreaks are if not explicitly against the terms of service, they are certainly implicitly against the terms of service and within OpenAI’s discretion to ban people for. Yet they appear to be letting people get away with a lot. I mean heck just the other day based on a HN comment on another topic, I whipped up an Jailbreak prompt to see how hard it is to get ChatGPT to flirt or be sexual, and how well it does at writing that kind of thing. Now sure it was just a one off experiment and while they were dealing with the history incident, but it’s not like it’s hard to have something flag the three letters “sex” and have some kind of telemetry, because they explicitly mention sexual content in the TOS… they aren’t targeting the low hanging fruit because it’s pointless. They’re letting the red team (the public) try and really test the boundaries to make sure that much more serious issues aren’t hiding, waiting for someone to find much later if they were more aggressively limiting the amount of implicit TOS violations that appear to be happening.
- sebzim4500 4y ago
  I think it's far more likely they just didn't care very much if people play with unreleased plugins before they were supposed to.
- thricegr8 4y ago
  Can you expound on this?

CGamesPlay 4y ago

I interpreted this to mean that the plugins are actually client-side software, so of course there are client-side checks to confirm the allowed usage. That's why I can use a client-side proxy to access them. Right?

jeremyjh 4y ago
No, the plugins are server-side software, but the server isn't checking if the user is allowed to use the plugin requested by the client.

gjsman-1000 4y ago

I think this is a perfect example about how you could be extremely smart in one field, and extremely dumb in another. You can have a programmer brilliant at AI who doesn't know the slightest thing about web app security.

rjzzleep 4y ago
Most don't. But many don't think that they should reinvent web security and that is the key difference here.
kodah 4y ago
This also translates on to Software Engineers when talking about Systems Engineering topics. People are quite silo'd and there's often people who have a specific overlap between two disciplines.
precompute 4y ago
I think instead of making excuses about this, it's rational to expect a billion dollar "non-profit" to hire a team of competent web devs / backend people for their product that is only accessible via the internet.

efficax 4y ago

maybe they asked chatgpt if it was secure and trusted the response

Cthulhu_ 4y ago

It feels like the people at OpenAI have been pushed hard to get these features out the door - with the release of ChatGPT, suddenly many big players have rushed to release their (GPT 3/4) based products, which will involve multi-billion contracts. If they had waited for things like security reviews, they might have missed the boat. Developments are going really fast in this space.

fisf 4y ago
If things like the issues we have seen in the last days require a security review in order to be caught, then I am not very optimistic about the rest of their stack.
- kingkawn 4y ago
  Don’t worry, we’ll have the AI secure it’s own code, it’ll go fiiiiine
pixl97 4y ago
"We were going to implement AI safety, but like the competitors are going to fast and we needed to release first" We are going to great filter ourselves.
- jacquesm 4y ago
  There is a fantastic bit in one of Terry Pratchett's books: "Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."
  - pixl97 4y ago
    Oh dear, that may be me. I was at Home Depot in the checkout line and they had some construction going on. Hanging down from the ceiling by wires was a red button. I was compelled, I could not avoid it. I pushed the red button. A loud buzzing noise occurred. The cashier sighed and said that I was the sixth guy to do that, not a single woman had pressed the button so far.

axg11 4y ago

I don't think _this_ is a security check. These are essentially unlisted plugins. They will be released/approved and this is a way to allow some user to test the plugins (different client side software).

ethanbond 4y ago
Have they been approved and released or not? If no, then they’re not meant to be accessible, meaning it’s a security flaw to be able to access them.
- IanCal 4y ago
  What exactly is the security issue? The thing that's affected here is the responses you alone see.
  - djbusby 4y ago
    They are saying that client-side only "ACL" is sloppy and that could be an indication of even more internal slop (of which title-leak may be another symptom)
    - londons_explore 4y ago
      I suspect it was a deliberate decision not to ACL plugins. They let anyone create and use one after all. The only reason approval exists at all is so users aren't tricked into running low quality or spammy plugins. You could consider this similar to someone revealing that lots of apps banned from app stores are available on other websites and one could write the headline "banned app leaks onto apkmirror.com, is google security compromised?"
      - ethanbond 4y ago
        No, you could consider this similar to someone revealing that lots of apps banned from app stores are available on the same app stores that they’re banned from, which, yeah, looks a bit dodgy.

jnwatson 4y ago

If you on-purpose bypass a filter to use an unapproved plugin, it is on the user at that point.

lavezzi 4y ago

Funny you say that because OpenAI just sent out an email saying "another user may have inadvertently seen your billing information when clicking on their own “Manage Subscription” page." for a nine hour window yesterday

cma 4y ago
They didn't have the billing system at launch, so maybe it was coded with the help of chatGPT.
jwestbury 4y ago
Ah, yes, the Valve-patented "let's cache user account pages in the CDN" strategy.
- _piif 4y ago
  Did they also have a similar incident before?
  - EthicalSimilar 4y ago
    Yes, a few years ago. It is the first thing I thought of when I saw this news. For a brief period of time (a few hours?), Steam would show you as being logged in as another user. You could view their billing page and other restricted pages.

yakz 4y ago

When I rewrite the features response and it then queries the plug-in list, the server responds with a 401. So maybe not purely client side (I have a plus account but no plug-in access yet).

magic_hamster 4y ago

If plugins are just text prefixes, which reside on the server and have been approved, then how is it more risky than accepting prompts in general?

salawat 4y ago

It isn't about security, it's about controlling the default baseline ecosystem. If it were truly about security A) it'd actually be secure B) it'd be way more user-hostile than a client-side ACL.

moneywoes 4y ago

Title information leak?

danShumway 4y ago

My conspiracy theory is that all of the "we take security very seriously" talk out of OpenAI has aligned with "our AI is so advanced and powerful that we have to take these existential risks very seriously and we have to make sure it doesn't turn evil." And OpenAI doesn't seem very interested in security conversations that don't feed into that narrative[0]. I think it's (mostly) just propaganda. OpenAI focuses on dangers that make their AI seem more advanced than it is, and ignore dangers that don't play well to the press (like clientside validation). I'm not even sure it's standard "we care about your security" company talk, I think it's mostly designed just to make people think of GPT as more cutting edge. The "GPT paid someone to solve a captcha" release seems like pure publicity stunt to me, given that solving captchas by pretending to be blind is something other systems can already do without human help. But it played well in press releases. ---- [0]: To be fair, they have been focusing on alignment, but I honestly don't think alignment is a security measure, I think it's a general performance measure. We haven't seen strong evidence that alignment can stop prompt injection attacks, so I think mostly OpenAI mostly just cares about the potential negative press from their AI randomly insulting people.

precompute 4y ago
OpenAI is 50% product, 50% advertising.
- tmpz22 4y ago
  * 50% product, 50% fundraising
  - naniwaduni 4y ago
    Compromise: 50% advertising, 50% fundraising?
- Balgair 4y ago
  I mean, that's a lot better than most businesses.
  - 4y ago

adambard 4y ago

So it turns out that the way ChatGPT plugins operate, any user can use unverified plugins from arbitrary urls, just by pasting in a URL: https://platform.openai.com/docs/plugins/introduction/plugin-flow https://platform.openai.com/docs/plugins/introduction/plugin... With this in mind, how concerned should we really be about a pre-registered list of unverified plugins? The only new information exposed is the URLs of these various plugins, is this information so private if it was already registered in their list to begin with?

antihero 4y ago

Perhaps the AI guys are insane as the Bitcoin guys

Palpatineli 4y ago

What can you expect from a company releasing a (at least) village idiot level AI and give it access to the internet AND a shell/Python interpreter with skin-deep alignment that is effortlessly circumvented with DAN?