7 ms·

Reported a flaw to OpenAI that lets users peek at others' chat responses. Got an auto-reply on May 29th, radio silence since. Issue remains unpatched :( Avoided

Reported a flaw to OpenAI that lets users peek at others' chat responses. Got an auto-reply on May 29th, radio silence since. Issue remains unpatched :( Avoided their bug bounty due to permanent NDAs preventing disclosure even after fixes. Following standard 45-day disclosure window—users should avoid sharing sensitive data until this is resolved.

fcpguru 1y ago
well done, sounds very reasonable and following the rules.
- requilence 1y ago
  Appreciate it. Just trying to do the right thing by both OpenAI and users here.
poniko 1y ago
The NDA part feels really murky.
- tptacek 1y ago
  It's pretty standard for bounty programs. If you don't like it, which is reasonable, do what this researcher did and just post independently.
  - pyman 1y ago
    The bug bounty world is a funny one. I remember one complaining that their bug was dismissed and fixed after they signed an NDA, no payout, nothing. Another one got $100 instead of $5,000 because the company downgraded the severity from high to low. So they ended up with little or no money, and no recognition either. Not sure if these were edge cases, but it does make you wonder how fair the process really is.
    - tptacek 1y ago
      If you're dealing with large companies, a good rule of thumb is that the bounty program is incentivized to pay you out. Their internal metrics improve the more they pay; the point is to turn up interesting bugs, and the figure of merit for that is "how much did we have to spend". At a large company, a bounty that isn't paying anything out is a failure. All bets are off with small random startups that do bug bounties because they think they're supposed to (most companies should not run bounties). But that's not OpenAI. Dave Aitel works at OpenAI. They're not trying to stiff you. Simultaneous discovery (either with other researchers or, even more often, with internal assessments) is super common. What's more, you're not going to get any corroboration or context for them (sets up a crazy bad incentive with bounty seekers, who litigate bounty results endlessly). When you get a weird and unfair-seeming response to a bounty from a big tech company, for the sake of your own sanity (and because you'll probably be right), just assume someone internal found the bug before you did, and you reported it in the (sometimes long) window during which they were fixing it.
      - pyman 1y ago
        Interesting insights, thanks for sharing
  - asadotzler 1y ago
    That's an exaggeration. Most industry leaders do not require NDAs, only coordinated disclosure. Mozilla's program, which has been around longer than most, doesn't. Google and Microsoft don't. Meta and Apple don't. This is water carrying, intentional or not, for a terrible practice that should be shamed, so that it doesn't become standard.
    - tptacek 1y ago
      My understanding is that all Bugcrowd bounties do by default. You can shame it all you want, but you can also just publish your bugs directly. Nobody has to use the Bugcrowd platform. You don't even have to wait 45 days; I don't buy these "CERT/CC" rules.
      - asadotzler 1y ago
        You said it was pretty standard for bug bounty programs, and I disagreed pointing to several of the largest and longest lived bug bounty programs, none of which do that, and your response is pointing out that one particular platform does it? Even among 3rd party platforms, of which there are several bigs, the NDAs are not a platform requirement, just an option for participating firms. NDAs are not the norm. Don't mislead people who would otherwise get into this game with non-issues they need not worry over.
        tptacek 1y ago
        OpenAI's security team commented on the thread themselves that they believe they simply accepted the Bugcrowd defaults. I think you're trying to find a controversy that just isn't here.
jonrouach 1y ago
you're sure it's not their "feature" that calling the api with empty string returns random hallucinations? https://jarbon.medium.com/gpt-prompt-bug-94322a96c574 https://jarbon.medium.com/gpt-prompt-bug-94322a96c574
- requilence 1y ago
  No, definitely not the empty string hallucination bug. These are clearly real user conversations. They start like proper replies to requests, sometimes reference the original question, and appear in different languages.
  - JyB 1y ago
    I don’t see anything here that would prevent a LLM from generating these. Right?
    - requilence 1y ago
      In one of the responses, it provided the financial analysis of a not well-known company with a non-Latin name located in a small country. I found this company; it is real and numbers in the response are real. When I asked my ChatGPT to provide a financial report for this company without using web tools, it responded: `Unfortunately, I don’t have specific financial statements for “xxx” for 2021 and 2022 in my training data, and since you’ve asked not to use web search, I can’t pull them live.`.
      - Sebguer 1y ago
        Do you understand what a hallucination is?
        jojobas 1y ago
        Coming up with accurate financial data that you can't get it to report outright doesn't seem like one.
maxlin 1y ago
Permanent NDA's? Oof. It's like their plan is to just try to force the lid down till they reach ASI or something lol
- tptacek 1y ago
  Again: NDAs are bog standard bounty terms.
999900000999 1y ago
Users should always avoid sharing sensitive data. A lot of AI products straight up have plan text logs available for everyone at the company to view.
- ameliaquining 1y ago
  Which ones? Do you just mean tiny startups and side projects and the like or is this a problem that major model providers have?
- pyman 1y ago
  It's not just about sensitive data like passwords, contracts, or IP. It's also about the personal conversations people have with ChatGPT. Some are depressed, some are dealing with bullying, others are trying to figure out how to come out to their parents. For them, this isn't just sensitive, it's life-changing if it gets leaked. It's like Meta leaking their WhatsApp messages. I really hope they fix this bug and start taking security more seriously. Trust is everything.
  - milkshakes 1y ago
    maybe you should stop trusting random people on the internet making extraordinary claims without proof then?
    - baby_souffle 1y ago
      Isn't "assume vulnerable" The only prudent thing to do here?
      - refulgentis 1y ago
        No? Yes? Mu? After some hemming and hawing, my most cromulent thought is, having good security posture isn't synonymous with accepting every claim you get from the firehose
      - milkshakes 1y ago
com2kid 1y ago
I see other users conversations on my Gemini dashboard, not sure who to even complain to. Software quality is... Minimal now days.
deleted 1y ago
[deleted]

fcpguru 1y ago

well done, sounds very reasonable and following the rules.

requilence 1y ago
Appreciate it. Just trying to do the right thing by both OpenAI and users here.

poniko 1y ago

The NDA part feels really murky.

tptacek 1y ago
It's pretty standard for bounty programs. If you don't like it, which is reasonable, do what this researcher did and just post independently.
- pyman 1y ago
  The bug bounty world is a funny one. I remember one complaining that their bug was dismissed and fixed after they signed an NDA, no payout, nothing. Another one got $100 instead of $5,000 because the company downgraded the severity from high to low. So they ended up with little or no money, and no recognition either. Not sure if these were edge cases, but it does make you wonder how fair the process really is.
  - tptacek 1y ago
    If you're dealing with large companies, a good rule of thumb is that the bounty program is incentivized to pay you out. Their internal metrics improve the more they pay; the point is to turn up interesting bugs, and the figure of merit for that is "how much did we have to spend". At a large company, a bounty that isn't paying anything out is a failure. All bets are off with small random startups that do bug bounties because they think they're supposed to (most companies should not run bounties). But that's not OpenAI. Dave Aitel works at OpenAI. They're not trying to stiff you. Simultaneous discovery (either with other researchers or, even more often, with internal assessments) is super common. What's more, you're not going to get any corroboration or context for them (sets up a crazy bad incentive with bounty seekers, who litigate bounty results endlessly). When you get a weird and unfair-seeming response to a bounty from a big tech company, for the sake of your own sanity (and because you'll probably be right), just assume someone internal found the bug before you did, and you reported it in the (sometimes long) window during which they were fixing it.
    - pyman 1y ago
      Interesting insights, thanks for sharing
- asadotzler 1y ago
  That's an exaggeration. Most industry leaders do not require NDAs, only coordinated disclosure. Mozilla's program, which has been around longer than most, doesn't. Google and Microsoft don't. Meta and Apple don't. This is water carrying, intentional or not, for a terrible practice that should be shamed, so that it doesn't become standard.
  - tptacek 1y ago
    My understanding is that all Bugcrowd bounties do by default. You can shame it all you want, but you can also just publish your bugs directly. Nobody has to use the Bugcrowd platform. You don't even have to wait 45 days; I don't buy these "CERT/CC" rules.
    - asadotzler 1y ago
      You said it was pretty standard for bug bounty programs, and I disagreed pointing to several of the largest and longest lived bug bounty programs, none of which do that, and your response is pointing out that one particular platform does it? Even among 3rd party platforms, of which there are several bigs, the NDAs are not a platform requirement, just an option for participating firms. NDAs are not the norm. Don't mislead people who would otherwise get into this game with non-issues they need not worry over.
      - tptacek 1y ago
        OpenAI's security team commented on the thread themselves that they believe they simply accepted the Bugcrowd defaults. I think you're trying to find a controversy that just isn't here.

jonrouach 1y ago

you're sure it's not their "feature" that calling the api with empty string returns random hallucinations? https://jarbon.medium.com/gpt-prompt-bug-94322a96c574 https://jarbon.medium.com/gpt-prompt-bug-94322a96c574

requilence 1y ago
No, definitely not the empty string hallucination bug. These are clearly real user conversations. They start like proper replies to requests, sometimes reference the original question, and appear in different languages.
- JyB 1y ago
  I don’t see anything here that would prevent a LLM from generating these. Right?
  - requilence 1y ago
    In one of the responses, it provided the financial analysis of a not well-known company with a non-Latin name located in a small country. I found this company; it is real and numbers in the response are real. When I asked my ChatGPT to provide a financial report for this company without using web tools, it responded: `Unfortunately, I don’t have specific financial statements for “xxx” for 2021 and 2022 in my training data, and since you’ve asked not to use web search, I can’t pull them live.`.
    - Sebguer 1y ago
      Do you understand what a hallucination is?
      - jojobas 1y ago
        Coming up with accurate financial data that you can't get it to report outright doesn't seem like one.

maxlin 1y ago

Permanent NDA's? Oof. It's like their plan is to just try to force the lid down till they reach ASI or something lol

tptacek 1y ago
Again: NDAs are bog standard bounty terms.

999900000999 1y ago

Users should always avoid sharing sensitive data. A lot of AI products straight up have plan text logs available for everyone at the company to view.

ameliaquining 1y ago
Which ones? Do you just mean tiny startups and side projects and the like or is this a problem that major model providers have?
pyman 1y ago
It's not just about sensitive data like passwords, contracts, or IP. It's also about the personal conversations people have with ChatGPT. Some are depressed, some are dealing with bullying, others are trying to figure out how to come out to their parents. For them, this isn't just sensitive, it's life-changing if it gets leaked. It's like Meta leaking their WhatsApp messages. I really hope they fix this bug and start taking security more seriously. Trust is everything.
- milkshakes 1y ago
  maybe you should stop trusting random people on the internet making extraordinary claims without proof then?
  - baby_souffle 1y ago
    Isn't "assume vulnerable" The only prudent thing to do here?
    - refulgentis 1y ago
      No? Yes? Mu? After some hemming and hawing, my most cromulent thought is, having good security posture isn't synonymous with accepting every claim you get from the firehose
    - milkshakes 1y ago

com2kid 1y ago

I see other users conversations on my Gemini dashboard, not sure who to even complain to. Software quality is... Minimal now days.

deleted 1y ago

[deleted]