3 ms·

These "harmful" responses seem completely useless; exploits are readily available on the net in the form of metasploit, vulnerability databases, etc. Pretty sta

by arp242 3y ago

These "harmful" responses seem completely useless; exploits are readily available on the net in the form of metasploit, vulnerability databases, etc. Pretty standard stuff. The counterfeit money response is even more useless; all it says is "copy real money" but in a really roundabout way. Well, who would have guessed? And these type of instructions are readily available on e.g. YouTube https://www.youtube.com/watch?v=xG6oCrtef5A https://www.youtube.com/watch?v=xG6oCrtef5A If you're savvy enough to do these type of prompt injection then you're savvy enough to get more useful responses from other sources, like this little obscure site called "Google".

xmcqdpt2 3y ago
The idea with these isn't that you can trick chat gpt into giving you bad descriptions of how to do crimes, but that when someone decides to wire it up to something with real world impact, you'll be able to push it towards something you want. For example, we are told that customer support is about to be fully automated soon. These attacks could be used to eg get refunds for bogus reasons. There is already one real life example I know of that didn't even need tricks, https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/amp/ https://www.forbes.com/sites/marisagarcia/2024/02/19/what-ai...
- UweSchmidt 3y ago
  The space of ChatGPT tricked for fun or fraud is probably just a subset of the error and confusion in which those AIs operate - generally to the confusion of the customers that they "support". I can imagine a future where humans have to deal with an uncanny valley of unreliable and confusing AIs of increasing significance that are wired up to more and more real world systems. And at the same time companies reserving the right to legally back out of any adverse outcomes through a fine-print EULA and unintelligible documentation in hard to find places.
- arp242 3y ago
  Well okay, but that's quite a different thing than "these are harmful responses".
andy99 3y ago
Yeah it has some minor academic interest but the idea of these exploits leading to something "harmful" is hyperbole to say the least. An interesting starting point would be to show that an uncensored LLM could synthesize knowledge that isn't already easily available to provide realistic instructions that could be followed by a non specialist without access to controlled anything to make or do some bad thing.
ecoquant 3y ago
"Harmful" has really become a ridiculous word in my lifetime. In Fortune's Formula by William Poundstone it mentions how MIT professor/Black Jack card counting/Hedge Fund manager Edward O. Thorp spent a period as a kid building pipe bombs and various chemical explosives for fun. That is what I could consider a harmful activity. To pretend ASCII art can be harmful strikes me as the thoughts of someone with paranoid delusions. Someone completely detached from reality that probably needs medication.
- andy99 3y ago
  It goes along with "safety". The point is that something becomes important and is agreed upon as worth attention so then suddenly people adopt that terminology to push their agenda through. See also bullying, human rights, etc.
  - andsoitis 3y ago
    > is agreed upon as worth attention That’s just it. There are many things that some group of people will say is harmful, but that many or even most others think is hyperbole.
- swozey 3y ago
  You post literally the most ignorant and hilarious comments I've ever read on HN. I'm glad you're shadowbanned, but it's unfortunate you're completely oblivious to that fact and keep commenting hour after hour with drivel that nobody reads. It also makes me confused as to why only some of your comments are dead flagged but not all. I guess I understand the HN shadowban less than I thought I did. The entire world is better hearing your opinion of the word "harmful" though, where would we be without your opinion?! You have such gems as "HTML Monkeys" - if there is any ageism in this industry it's caused by miserable old men like you who absolutely nobody wants to work with. It's absolutely fitting that your knowledge seems to only contain "Perl" and "C." Welcome to 2024, and I know full well you're not a kernel dev, because I am, and we wouldn't use C and Perl in the same sentence. You're the equivalent to me of my ignorant, dead, racist grandfather going on Facebook and ranting about Jews or Muslims except replace that ignorance with your technology skillset. That's how important your commenting is to probably a majority of HN users who actually work in high end technology in 2024. Enjoy your $85k Perl job while we make $350k writing rust and go and being "basic programmers who don't know programming" because we started writing HTML for our myspace pages at 12 which was evidently beneath you. And now we're launching rockets and building distributed systems when you.. what, made a static web forum in Perl that probably had absolutely no security considerations, was single-threaded (LOL) and absolutely did not autoscale or self-repair? You can grow old and not become a miserable, rude, depressing old man. It's possible. Also, keep your skills up to date. Nobody, literally nobody talks about perl aside from $35k/yr Russian devs in 2024. I'm 40. I'm not 20. I've been in tech since I was 10. You don't know more than everyone else here because you're old. We were there too - we were just 12, not 30. And stop using the term "HTML Monkey" for fucks sake. It's incredibly derogatory. You probably use "cable monkey" and a bunch of other "monkey" names. They are all derogatory. Be a better person. https://news.ycombinator.com/context?id=39627175 https://news.ycombinator.com/context?id=39627175

by arp242 3y ago

xmcqdpt2 3y ago

The idea with these isn't that you can trick chat gpt into giving you bad descriptions of how to do crimes, but that when someone decides to wire it up to something with real world impact, you'll be able to push it towards something you want. For example, we are told that customer support is about to be fully automated soon. These attacks could be used to eg get refunds for bogus reasons. There is already one real life example I know of that didn't even need tricks, https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/amp/ https://www.forbes.com/sites/marisagarcia/2024/02/19/what-ai...

UweSchmidt 3y ago
The space of ChatGPT tricked for fun or fraud is probably just a subset of the error and confusion in which those AIs operate - generally to the confusion of the customers that they "support". I can imagine a future where humans have to deal with an uncanny valley of unreliable and confusing AIs of increasing significance that are wired up to more and more real world systems. And at the same time companies reserving the right to legally back out of any adverse outcomes through a fine-print EULA and unintelligible documentation in hard to find places.
arp242 3y ago
Well okay, but that's quite a different thing than "these are harmful responses".

andy99 3y ago

Yeah it has some minor academic interest but the idea of these exploits leading to something "harmful" is hyperbole to say the least. An interesting starting point would be to show that an uncensored LLM could synthesize knowledge that isn't already easily available to provide realistic instructions that could be followed by a non specialist without access to controlled anything to make or do some bad thing.

ecoquant 3y ago

"Harmful" has really become a ridiculous word in my lifetime. In Fortune's Formula by William Poundstone it mentions how MIT professor/Black Jack card counting/Hedge Fund manager Edward O. Thorp spent a period as a kid building pipe bombs and various chemical explosives for fun. That is what I could consider a harmful activity. To pretend ASCII art can be harmful strikes me as the thoughts of someone with paranoid delusions. Someone completely detached from reality that probably needs medication.

andy99 3y ago
It goes along with "safety". The point is that something becomes important and is agreed upon as worth attention so then suddenly people adopt that terminology to push their agenda through. See also bullying, human rights, etc.
- andsoitis 3y ago
  > is agreed upon as worth attention That’s just it. There are many things that some group of people will say is harmful, but that many or even most others think is hyperbole.
swozey 3y ago
You post literally the most ignorant and hilarious comments I've ever read on HN. I'm glad you're shadowbanned, but it's unfortunate you're completely oblivious to that fact and keep commenting hour after hour with drivel that nobody reads. It also makes me confused as to why only some of your comments are dead flagged but not all. I guess I understand the HN shadowban less than I thought I did. The entire world is better hearing your opinion of the word "harmful" though, where would we be without your opinion?! You have such gems as "HTML Monkeys" - if there is any ageism in this industry it's caused by miserable old men like you who absolutely nobody wants to work with. It's absolutely fitting that your knowledge seems to only contain "Perl" and "C." Welcome to 2024, and I know full well you're not a kernel dev, because I am, and we wouldn't use C and Perl in the same sentence. You're the equivalent to me of my ignorant, dead, racist grandfather going on Facebook and ranting about Jews or Muslims except replace that ignorance with your technology skillset. That's how important your commenting is to probably a majority of HN users who actually work in high end technology in 2024. Enjoy your $85k Perl job while we make $350k writing rust and go and being "basic programmers who don't know programming" because we started writing HTML for our myspace pages at 12 which was evidently beneath you. And now we're launching rockets and building distributed systems when you.. what, made a static web forum in Perl that probably had absolutely no security considerations, was single-threaded (LOL) and absolutely did not autoscale or self-repair? You can grow old and not become a miserable, rude, depressing old man. It's possible. Also, keep your skills up to date. Nobody, literally nobody talks about perl aside from $35k/yr Russian devs in 2024. I'm 40. I'm not 20. I've been in tech since I was 10. You don't know more than everyone else here because you're old. We were there too - we were just 12, not 30. And stop using the term "HTML Monkey" for fucks sake. It's incredibly derogatory. You probably use "cable monkey" and a bunch of other "monkey" names. They are all derogatory. Be a better person. https://news.ycombinator.com/context?id=39627175 https://news.ycombinator.com/context?id=39627175