7 ms·

OpenAI CEO 'feels awful' about ChatGPT leaking conversations

44 points by LinuxBender 4y ago

doodlesdev 4y ago
> we had a significant issue in ChatGPT due to a bug in an open source library, for which a fix has now been released and we have just finished validating. WHICH open source library? Seems like they fucked up and are looking for a scapegoat. There's no excuses for issues like this. Also, I believe this is yet again one more reason for companies and developers to consider row-level security [0], it simply erases the possibility of a malformed query returning data that shouldn't be accessible altogether. [0]: https://www.postgresql.org/docs/current/ddl-rowsecurity.html https://www.postgresql.org/docs/current/ddl-rowsecurity.html
- dilippkumar 4y ago
  From their blog [0] The bug was discovered in the Redis client open-source library, redis-py. As soon as we identified the bug, we reached out to the Redis maintainers with a patch to resolve the issue. Here’s how the bug worked: - We use Redis to cache user information in our server so we don’t need to check our database for every request. - We use Redis Cluster to distribute this load over multiple Redis instances. - We use the redis-py library to interface with Redis from our Python server, which runs with Asyncio. - The library maintains a shared pool of connections between the server and the cluster, and recycles a connection to be used for another request once done. - When using Asyncio, requests and responses with redis-py behave as two queues: the caller pushes a request onto the incoming queue, and will pop a response from the outgoing queue, and then return the connection to the pool. - If a request is canceled after the request is pushed onto the incoming queue, but before the response popped from the outgoing queue, we see our bug: the connection thus becomes corrupted and the next response that’s dequeued for an unrelated request can receive data left behind in the connection. - In most cases, this results in an unrecoverable server error, and the user will have to try their request again. - But in some cases the corrupted data happens to match the data type the requester was expecting, and so what gets returned from the cache appears valid, even if it belongs to another user. At 1 a.m. Pacific time on Monday, March 20, we inadvertently introduced a change to our server that caused a spike in Redis request cancellations. This created a small probability for each connection to return bad data. This bug only appeared in the Asyncio redis-py client for Redis Cluster, and has now been fixed. [0] https://openai.com/blog/march-20-chatgpt-outage https://openai.com/blog/march-20-chatgpt-outage
  - merek 4y ago
    Thanks for the summary.
  - doodlesdev 4y ago
    A few questions this still doesn't answer: why was there credit card data in their Redis cluster, and why were they the only ones affected that I know of? Still, this response seems much more reasonable than what I was expecting from OpenAI considering the seriousness of the issue. Anyways, thank you very much for the summary, I didn't find this blog post in the publications that were talking about the issue.
    - deleted 4y ago
      [deleted]
  - jonny_eh 4y ago
    A think a reasonable lesson from this is to double check that the returned data belongs to the user you're about to serve it to. So include some way to associate the owning user with each cache entry, so it can be verified later when it's recalled.
- bspammer 4y ago
  Wouldn’t this require creating a database role for every user? Then your backend needs to be able to assume every user role, which still seems to allow for bugs.
  - jansommer 4y ago
    This seems to be a common misconception. You do not need a role for every user. You can assign session variables, at least in Postgres, and use them in your policies.
- adql 4y ago
  > Also, I believe this is yet again one more reason for companies and developers to consider row-level security [0], it simply erases the possibility of a malformed query returning data that shouldn't be accessible altogether. That requires 1:1 DB user to real user mapping + one connection to DB per users, it is entirely horrible performance-wise, which is why it is so rarely used for anything more than separating apps from eachother.
  - jansommer 4y ago
    No it doesn't. No need for 1:1 db user mapping.
whateveracct 4y ago
Bugs in your code are bugs in your code. If they are user-facing, you could've had tests to catch them.
- sharemywin 4y ago
  this.
- curiousgal 4y ago
  But it's an open-source library... /s
  - moffkalast 4y ago
    Maybe they should be reminded that redis-py is under the MIT license. > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- nawgz 4y ago
  "Why didn't you simply have a test for an unexpected failure case which arose as a result of moving pieces across a distributed system?" - whateveracct
  - aldarisbm 4y ago
    you’re retrieving data from a cache, is it so outrageous to say that they should confirm that the data belongs to the account is retrieving it?
    - Atheros 4y ago
      It is pretty outrageous, yes. If Alice logs in and I correctly request Alice's details from the database, it's reasonable for me to expect that the database either returns Alice's details or an error. It shouldn't be up to application-layer programmers to code checks to make sure that a database doesn't return the wrong data 1% of the time. That kind of test is a test for the developers of Asyncio and Redis to implement, not users of Asyncio and Redis.
gsatic 4y ago
Did you guys see the Trump chat where he asks it how to take over the country?
- dgellow 4y ago
  Do you have a link, or a source?
andrewstuart 4y ago
It's telling of character that Sam doesn't take responsibility, and blames it on an open source package in literally the same breath. OpenAI is a company that we are going to have a significant relationship with into the future, so it's worth understanding the characteristics of the leadership of that company. A real leader would simply have said "we fixed it, sorry", instead of "we fixed it, but it was someone else's fault". It reminds me of those "I deleted the production database and got fired" stories ..... I always think it's management to blame if it is possible to delete the production database, but management like to blame someone further down the tree.
- joshxyz 4y ago
  I'm confused, why pin it on the guy when it's really caused by an open source library? Do you want full transparency or what? Jesus.
  - arthurcolle 4y ago
    Anyone have the PR for the code change to fix this "bug" ?
    - circuit10 4y ago
      https://github.com/redis/redis-py/pull/2641 https://github.com/redis/redis-py/pull/2641
      - eurasiantiger 4y ago
        What happens if some exception other than CancelledError is thrown?
        arthurcolle 4y ago
        Segmentation fault! (core dumped)
  - bradDonniger 4y ago
DanielVZ 4y ago
On the one hand I feel empathy towards the engineers because I know third party bugs can feel somewhat unfair and obscure. On the other hand if I understand correctly, the bug arises from an edge case from an error that was very common. I constantly receive these errors using the chat interface, and that’s an organization issue. There should be an SRE team or someone to notice them and try to fix them.
paxys 4y ago
The issue was relatively minor in the grand scheme of things, but the "it was a bug in an open source library, so not our fault" attitude from the CEO, especially when addressing a user data leak, is concerning. Instead of blame deflection, the communication users (and companies) want to hear is – "it was our fault, and we are overhauling our security and vendor audit process to make sure it doesn't happen again."
scohesc 4y ago
It sounds exactly like the Silicon Valley excuse we all know and love. Sorry your account's been shut down, it must have been the silly algorithm/silly source code/silly engineers in the workshop again! When you're making a tool that could quite likely change the course of how people use and interact with computers, these excuses don't fly. Hope it doesn't happen again, but the old mantra of "move fast, break things, apologize later" seems to apply here just like with every other SV company.
- sebzim4500 4y ago
  >the old mantra of "move fast, break things, apologize later" seems to apply here just like with every other SV company. I mean, obviously? They would not be able to ship things at this speed with so few employees if they were being conservative. No value judgement, just saying we should not be surprised.
1B05H1N 4y ago
They're going to end up fumbling their momentum with the bad PR/optics.
flaque 4y ago
This article is poorly written, OpenAI has an actual postmortem here: https://openai.com/blog/march-20-chatgpt-outage https://openai.com/blog/march-20-chatgpt-outage.
nullsense 4y ago
If he didn't, we should either not trust him or be very unhappy he's in that job.
deleted 4y ago
[deleted]
favaq 4y ago
[flagged]
- andrewstuart 4y ago
  @dang
  - dang 4y ago
    That doesn't work. If you want to contact us, the only reliable way is to email hn@ycombinator.com.
- dang 4y ago
  Fine, I imagine? (You can't do this here, though, and I've banned the account.)
konfusinomicon 4y ago
feels awful but i bet the dopamine hit the engineer got from finding and fixing the bug felt fantastic
lee101 4y ago
[dead]

44 points by LinuxBender 4y ago

> we had a significant issue in ChatGPT due to a bug in an open source library, for which a fix has now been released and we have just finished validating. WHICH open source library? Seems like they fucked up and are looking for a scapegoat. There's no excuses for issues like this. Also, I believe this is yet again one more reason for companies and developers to consider row-level security [0], it simply erases the possibility of a malformed query returning data that shouldn't be accessible altogether. [0]: https://www.postgresql.org/docs/current/ddl-rowsecurity.html https://www.postgresql.org/docs/current/ddl-rowsecurity.html

dilippkumar 4y ago
From their blog [0] The bug was discovered in the Redis client open-source library, redis-py. As soon as we identified the bug, we reached out to the Redis maintainers with a patch to resolve the issue. Here’s how the bug worked: - We use Redis to cache user information in our server so we don’t need to check our database for every request. - We use Redis Cluster to distribute this load over multiple Redis instances. - We use the redis-py library to interface with Redis from our Python server, which runs with Asyncio. - The library maintains a shared pool of connections between the server and the cluster, and recycles a connection to be used for another request once done. - When using Asyncio, requests and responses with redis-py behave as two queues: the caller pushes a request onto the incoming queue, and will pop a response from the outgoing queue, and then return the connection to the pool. - If a request is canceled after the request is pushed onto the incoming queue, but before the response popped from the outgoing queue, we see our bug: the connection thus becomes corrupted and the next response that’s dequeued for an unrelated request can receive data left behind in the connection. - In most cases, this results in an unrecoverable server error, and the user will have to try their request again. - But in some cases the corrupted data happens to match the data type the requester was expecting, and so what gets returned from the cache appears valid, even if it belongs to another user. At 1 a.m. Pacific time on Monday, March 20, we inadvertently introduced a change to our server that caused a spike in Redis request cancellations. This created a small probability for each connection to return bad data. This bug only appeared in the Asyncio redis-py client for Redis Cluster, and has now been fixed. [0] https://openai.com/blog/march-20-chatgpt-outage https://openai.com/blog/march-20-chatgpt-outage
- merek 4y ago
  Thanks for the summary.
- doodlesdev 4y ago
  A few questions this still doesn't answer: why was there credit card data in their Redis cluster, and why were they the only ones affected that I know of? Still, this response seems much more reasonable than what I was expecting from OpenAI considering the seriousness of the issue. Anyways, thank you very much for the summary, I didn't find this blog post in the publications that were talking about the issue.
  - deleted 4y ago
    [deleted]
- jonny_eh 4y ago
  A think a reasonable lesson from this is to double check that the returned data belongs to the user you're about to serve it to. So include some way to associate the owning user with each cache entry, so it can be verified later when it's recalled.
bspammer 4y ago
Wouldn’t this require creating a database role for every user? Then your backend needs to be able to assume every user role, which still seems to allow for bugs.
- jansommer 4y ago
  This seems to be a common misconception. You do not need a role for every user. You can assign session variables, at least in Postgres, and use them in your policies.
adql 4y ago
> Also, I believe this is yet again one more reason for companies and developers to consider row-level security [0], it simply erases the possibility of a malformed query returning data that shouldn't be accessible altogether. That requires 1:1 DB user to real user mapping + one connection to DB per users, it is entirely horrible performance-wise, which is why it is so rarely used for anything more than separating apps from eachother.
- jansommer 4y ago
  No it doesn't. No need for 1:1 db user mapping.

whateveracct 4y ago

Bugs in your code are bugs in your code. If they are user-facing, you could've had tests to catch them.

sharemywin 4y ago
this.
curiousgal 4y ago
But it's an open-source library... /s
- moffkalast 4y ago
  Maybe they should be reminded that redis-py is under the MIT license. > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
nawgz 4y ago
"Why didn't you simply have a test for an unexpected failure case which arose as a result of moving pieces across a distributed system?" - whateveracct
- aldarisbm 4y ago
  you’re retrieving data from a cache, is it so outrageous to say that they should confirm that the data belongs to the account is retrieving it?
  - Atheros 4y ago
    It is pretty outrageous, yes. If Alice logs in and I correctly request Alice's details from the database, it's reasonable for me to expect that the database either returns Alice's details or an error. It shouldn't be up to application-layer programmers to code checks to make sure that a database doesn't return the wrong data 1% of the time. That kind of test is a test for the developers of Asyncio and Redis to implement, not users of Asyncio and Redis.

gsatic 4y ago

Did you guys see the Trump chat where he asks it how to take over the country?

dgellow 4y ago
Do you have a link, or a source?

andrewstuart 4y ago

It's telling of character that Sam doesn't take responsibility, and blames it on an open source package in literally the same breath. OpenAI is a company that we are going to have a significant relationship with into the future, so it's worth understanding the characteristics of the leadership of that company. A real leader would simply have said "we fixed it, sorry", instead of "we fixed it, but it was someone else's fault". It reminds me of those "I deleted the production database and got fired" stories ..... I always think it's management to blame if it is possible to delete the production database, but management like to blame someone further down the tree.

joshxyz 4y ago
I'm confused, why pin it on the guy when it's really caused by an open source library? Do you want full transparency or what? Jesus.
- arthurcolle 4y ago
  Anyone have the PR for the code change to fix this "bug" ?
  - circuit10 4y ago
    https://github.com/redis/redis-py/pull/2641 https://github.com/redis/redis-py/pull/2641
    - eurasiantiger 4y ago
      What happens if some exception other than CancelledError is thrown?
      - arthurcolle 4y ago
        Segmentation fault! (core dumped)
- bradDonniger 4y ago

DanielVZ 4y ago

On the one hand I feel empathy towards the engineers because I know third party bugs can feel somewhat unfair and obscure. On the other hand if I understand correctly, the bug arises from an edge case from an error that was very common. I constantly receive these errors using the chat interface, and that’s an organization issue. There should be an SRE team or someone to notice them and try to fix them.

paxys 4y ago

The issue was relatively minor in the grand scheme of things, but the "it was a bug in an open source library, so not our fault" attitude from the CEO, especially when addressing a user data leak, is concerning. Instead of blame deflection, the communication users (and companies) want to hear is – "it was our fault, and we are overhauling our security and vendor audit process to make sure it doesn't happen again."

scohesc 4y ago

It sounds exactly like the Silicon Valley excuse we all know and love. Sorry your account's been shut down, it must have been the silly algorithm/silly source code/silly engineers in the workshop again! When you're making a tool that could quite likely change the course of how people use and interact with computers, these excuses don't fly. Hope it doesn't happen again, but the old mantra of "move fast, break things, apologize later" seems to apply here just like with every other SV company.

sebzim4500 4y ago
>the old mantra of "move fast, break things, apologize later" seems to apply here just like with every other SV company. I mean, obviously? They would not be able to ship things at this speed with so few employees if they were being conservative. No value judgement, just saying we should not be surprised.

1B05H1N 4y ago

They're going to end up fumbling their momentum with the bad PR/optics.

flaque 4y ago

This article is poorly written, OpenAI has an actual postmortem here: https://openai.com/blog/march-20-chatgpt-outage https://openai.com/blog/march-20-chatgpt-outage.

nullsense 4y ago

If he didn't, we should either not trust him or be very unhappy he's in that job.

deleted 4y ago

[deleted]

favaq 4y ago

[flagged]

andrewstuart 4y ago
@dang
- dang 4y ago
  That doesn't work. If you want to contact us, the only reliable way is to email hn@ycombinator.com.
dang 4y ago
Fine, I imagine? (You can't do this here, though, and I've banned the account.)

konfusinomicon 4y ago

feels awful but i bet the dopamine hit the engineer got from finding and fixing the bug felt fantastic

lee101 4y ago

[dead]