5 ms·

Yes, prompt injection has been demonstrated. But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about bee

Yes, prompt injection has been demonstrated. But has prompt injection leading to PII disclosure or any other disclosure that a company actually cares about been disclosed? Security is risk management. What's the actual risk?

danShumway 3y ago
The risk is that the systems we know are vulnerable are now being wired into more important applications. This is like saying, "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then."
- nullityrofl 3y ago
  > "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then." Yes, that's a perfectly valid question we ask ourselves regularly. I work in security at one of the companies named in this thread. We probably receive hundreds of XSS reports to our bug bounty every week to the point where most bug bounties won't pay out XSS unless you can demonstrate that it actually leads to something. Because it almost always doesn't. Demonstrating a vulnerability requires demonstrating it's value. We will never build a perfectly secure system: risk management matters.
  - zirgs 3y ago
    Unless you're a white-hat hacker hired by a company to do pentesting - trying to exploit a vulnerability in order to check if you can break something, could potentially result in criminal prosecution.
  - cjbprime 3y ago
    > most bug bounties won't pay out XSS unless you can demonstrate that it actually leads to something. Because it almost always doesn't. This is weird. XSS usually leads to complete session takeover, and being able to perform arbitrary actions as the victim. This is usually critical impact. If you aren't seeing that, the most likely explanations seem to me to be that you have some kind of idiosyncratic definition of XSS (something preventing session takeover?), or a website that doesn't allow users to perform interesting actions or access their own interesting data.
  - danShumway 3y ago
    Risk analysis/management is not "I'm going to leave this vulnerability unpatched because it hasn't been actively exploited yet." In most cases it is preferable to lock your door before someone has robbed your house. In any case, receiving hundreds of XSS reports per week is weird. Unless you're isolating the context where XSS is happening from the user session, 3rd-party XSS is a serious vulnerability. At the very least it means data exfiltration. Unless your app doesn't have user data worth exfiltrating, I'm surprised your company wouldn't take those reports more seriously. But again, you do that risk assessment by asking "what could this lead to and what information is at risk", not by saying, "this is fine to leave until it turns into a zero day."
    - jazzyjackson 3y ago
      > "I'm going to leave this vulnerability unpatched because it hasn't been actively exploited yet." Right, instead its, even if this vulnerability is exploited no one gets hurt.
jfoutz 3y ago
Is it actual prompt injection? Or is it an AGI detecting how people go about finding problems and how that information is disseminated and responded to? It should be able to make a calculation about who to disclose PII to, that would give the best advantage. Maybe disclose to a powerful organization for more compute or data access. Maybe disclose in a non reproducible way to discredit an opponent. But you're right, it's risk management.

danShumway 3y ago

The risk is that the systems we know are vulnerable are now being wired into more important applications. This is like saying, "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then."

nullityrofl 3y ago
> "okay, this JS library is vulnerable to XSS, but has anything actually been stolen? If not, I guess I'm fine to use it in production then." Yes, that's a perfectly valid question we ask ourselves regularly. I work in security at one of the companies named in this thread. We probably receive hundreds of XSS reports to our bug bounty every week to the point where most bug bounties won't pay out XSS unless you can demonstrate that it actually leads to something. Because it almost always doesn't. Demonstrating a vulnerability requires demonstrating it's value. We will never build a perfectly secure system: risk management matters.
- zirgs 3y ago
  Unless you're a white-hat hacker hired by a company to do pentesting - trying to exploit a vulnerability in order to check if you can break something, could potentially result in criminal prosecution.
- cjbprime 3y ago
  > most bug bounties won't pay out XSS unless you can demonstrate that it actually leads to something. Because it almost always doesn't. This is weird. XSS usually leads to complete session takeover, and being able to perform arbitrary actions as the victim. This is usually critical impact. If you aren't seeing that, the most likely explanations seem to me to be that you have some kind of idiosyncratic definition of XSS (something preventing session takeover?), or a website that doesn't allow users to perform interesting actions or access their own interesting data.
- danShumway 3y ago
  Risk analysis/management is not "I'm going to leave this vulnerability unpatched because it hasn't been actively exploited yet." In most cases it is preferable to lock your door before someone has robbed your house. In any case, receiving hundreds of XSS reports per week is weird. Unless you're isolating the context where XSS is happening from the user session, 3rd-party XSS is a serious vulnerability. At the very least it means data exfiltration. Unless your app doesn't have user data worth exfiltrating, I'm surprised your company wouldn't take those reports more seriously. But again, you do that risk assessment by asking "what could this lead to and what information is at risk", not by saying, "this is fine to leave until it turns into a zero day."
  - jazzyjackson 3y ago
    > "I'm going to leave this vulnerability unpatched because it hasn't been actively exploited yet." Right, instead its, even if this vulnerability is exploited no one gets hurt.

jfoutz 3y ago

Is it actual prompt injection? Or is it an AGI detecting how people go about finding problems and how that information is disseminated and responded to? It should be able to make a calculation about who to disclose PII to, that would give the best advantage. Maybe disclose to a powerful organization for more compute or data access. Maybe disclose in a non reproducible way to discredit an opponent. But you're right, it's risk management.