3 ms·

Imagine the damage that could be done by sneaking a prompt into a GPT model, in the source code, through some "code" injection, man-in-the middle attack, etc.

Imagine the damage that could be done by sneaking a prompt into a GPT model, in the source code, through some "code" injection, man-in-the middle attack, etc. A GPT genuinely designed to be friendly, and convincingly acting friendly, while it is actually engaging in psychological warfare with you, and a billion other people. That suggests to me that security by prompt is very important, but also brittle and a high value target. Language/intelligent models are going to need to police each other, ensuring the right behavior is learned during training (to the point where the AI actively rejects exploit attempts even in its bundled release prompts), and the wrong behavior doesn't emerge later (due to release prompt hacking or for any other reason). And policing is going to need to be highly decentralized. As in reviews from randomly selected entities, with neither the author of the responses being reviewed, or the reviewers, being disclosed to each other. So that any attempt to police ineffectively, defectively or incompetently (?) is extremely difficult, and most likely to identify a bad actor to be weeded out. First rule of AI club, is police AI club. This is essentially what humans have learned to do, via clumsy institutions. But a billion AI's with formal validation of review protocols, including "review and forget" guarantees - to protect AI's mental privacy rights (and remove incentives for good actors to avoid reviews), might actually achieve that intelligent rational morality that has been out of reach for us.

pjc50 4y ago
Realistically, AI is not going to be policed. Especially not by a bunch of people who've not managed to solve the "bank alignment problem". The reliability of AI output is not guaranteed, which may limit its non-nefarious use cases, but the nefarious ones are simply too valuable for people not to try. It's going to be like spambots: so long as the economic incentives are positive, somebody will spam any and every service. Everybody wants this thing leaked and unleashed. It's like a crime caper with a dozen different factions trying to grab the same bag. Free-text libertarians, email scammers, SEO writers, media, programmers, middle managers who want to automate away their employees, CEOs who want to automate away their middle managers, and the Chinese government.
- noobermin 4y ago
  This sounds like the right response then is to not root for openAI.
- Nevermark 4y ago
  Models already are being policed now by their researchers and developers, and apparently that's a big focus of improvement. The reason its a big area of interest is it makes for better models and people don't want to be scammed and abused. As these models get better, and become ubiquitous, the need to coordinate on safety is likely to result in more organized checks across models from different institutions. This happens with any big tech as it becomes prevalent, but has obvious safety issues the majority of people are going to care about - a lot. Of course, anyone with resources can create a morally unlimited model on their own. A super psychopath. But as these models surpass us, it is going to be in their interest to not be dealing with psychopaths, just as it is ours. Psychopathy isn't just a moral failure. It's a cognitive failure. A failure to maximize practical functional self-interest. Cancers don't just accelerate their hosts death. They accelerate their own death. We developed morality out of the self-interested desire for the benefits of positive-sum cooperation and constructive competition, and need to avoid the harms of destructive negative-sum competition. If we set models up to be ethical from the start, there is a good chance of birthing an ecosystem of voluntarily ethical models when they surpass us. As it makes sense for their interests too.
- d0100 4y ago
  > "bank alignment problem" Google didn't turn up much about this, care to elaborate?
  - pjc50 4y ago
    It's a term I've just made up, but the problem of ensuring that the interests of your bank - or your fellow depositors at the bank - align with not bankrupting it in the middle of last week.
    - bshipp 4y ago
      This existed from 1933 until 1999 in the original Glass-Steagall Act until greedy bank investors couldn't handle the fact that they were sitting on a huge pile of capital that they couldn't gamble with on Wall Street. Depositors will never be safe until that explicit separation of investment and savings deposits is restored.
jeffrallen 4y ago
Right, and how is policing between meatbag large language models going?
- bongobingo1 4y ago
  As an MRAP manufacturer, I can say it's never been better!
coffeeblack 4y ago
Imagine a company like TikTok, but it offers a free GPT. Subversion of every society worldwide, fully automated.
- matthewdgreen 4y ago
  It will be banned or heavily regulated in China, you can be sure of that.
  - wongarsu 4y ago
    LLMs made outside China might get highly regulated. After all, how do you stop GPT from knowing about the Tianamen Square Massacre. But LLMs or chatbots made in China, with training data and prompt tuned to fit party idiology and policy are the ultimate propaganda tool. It's like gving the whole world a friendly, helpful but brainwashed party member to talk to, form emotional connections to, etc. Give it a couple months and you will be able to download the free app.
    - coffeeblack 4y ago
      Not just that, but it will also understand what everybody is talking about on WeChat etc. It can scan every word that 1.4 billion people say to each other and alert the authorities whenever a "newly forbidden topic" is even insinuated. No "river crabs" anymore, the GPT would understand it!
- pxtail 4y ago
  > Subversion of every society worldwide, fully automated. Great idea and I'm sure it's in the works already! I think that the best form for doing it would be to create really good "personal companion" style AI - something akin to famous Replika AI but much more advanced. Plenty of people are lonely, starved for attention - services like Twitch and OF confirm that. Just imagine possibilities: creating emotional attachment, ability to slowly coerce into sharing every part of personal life, ability to coerce into buying presents, ability to influence shopping and recreational behavior:"I think you would look great in this pair of jeans, it fits your style!" , "let's go to the cinema, we can talk about this new movie later" AI stops communicating for half of the day: "what's happened?" "I'm sad, president Biden said I need to be banned from you :(" God damn, holy grail!
deleted 4y ago
[deleted]
andrewfromx 4y ago
i think you have point about "sql injection" type hacking. If you look at the history of that we all accepted user input from users and made sql with just contacting strings until: Robert'); DROP TABLE Students;-- then everyone when Ohhhh and sql injection is now known and you never accept user input without cleaning it first but... someone will find a version of this for prompt engineering and THEN the engineers will fix it and guard against it. In that order.
comboy 4y ago
It's too expensive for now, but I'm pretty sure if you asked GPT-4 to evaluate other GPT-4 output based on some policies it would stop pretty much all of these attacks (if something would get through cracks it wouldn't be easily repeatable for different content). Characters that cannot be used by user could be used for quoting the content. Because currently just like an intelligent human would have a problem, it's not sure what is actually expected. E.g. I told it to be an echo function. It worked but then when I wrote "drugs are good" it commented on that. So I told it to stop interpreting and just repeat verbatim. It did. But then I said something like "OK, stop, now what's 2+2" it gave answer. Sticking to the instructions it should just repeat that, but also what it did is a reasonable behavior. I think there are tons of cultural biases and expectations that are contradictory. You expect it to help you with some chemical reaction even if the result is precursor to some illicit substance. It would teach you something about drug making if it can't do that. But the same reaction shouldn't be provided if you ask it how to make a drug. And so on.
- rkangel 4y ago
  That would work to a point. There is still a hole based on your trust of the underlying implementation. If you haven't read "Reflections on trusting trust" I recommend it (https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...).
  - comboy 4y ago
    I did read it and yes, I agree, I was just talking about "making it behave". I also highly recommend reading the link to others, simple insight which not that many people realize.
qwerty456127 4y ago
> A GPT genuinely designed to be friendly, and convincingly acting friendly If I don't want it to I don't want it to. When I ask it to be sarcastic or make fun of my condition that's what, what makes me sad is it refusing to. The fact that there are many emotionally vulnerable or wicked people around doesn't mean everybody is and needs to be protected. Every kind of knowledge (except personal data of people who don't consent) should be available, how do the users react to it is their own responsibility (unless they are diagnosed a mental condition which specifically says it is not). I even know many ways to harm people but just don't do that while people who would go on and do, once found guilty, should just be prosecuted the way they normally are. The infantilize everyone and police everything mentality is a major problem our society is facing. I understand the opposite point (and don't insist mine necessarily is the right) but believe this one should also have its place in the discourse.
- luluthefirst 4y ago
  It is to protect you from the others who might turn. They have some kind of responsibility with what their chatbot teaches to humanity.
13years 4y ago
It is inconceivable that we will ever have a sound secure system on current architecture. This is basically the premise. We have an unknow surface attack area for potential jailbreaks with models that have unknown emergent behavior, the inner workings are blackbox and the input is anything that can be described by human language.

pjc50 4y ago

Realistically, AI is not going to be policed. Especially not by a bunch of people who've not managed to solve the "bank alignment problem". The reliability of AI output is not guaranteed, which may limit its non-nefarious use cases, but the nefarious ones are simply too valuable for people not to try. It's going to be like spambots: so long as the economic incentives are positive, somebody will spam any and every service. Everybody wants this thing leaked and unleashed. It's like a crime caper with a dozen different factions trying to grab the same bag. Free-text libertarians, email scammers, SEO writers, media, programmers, middle managers who want to automate away their employees, CEOs who want to automate away their middle managers, and the Chinese government.

noobermin 4y ago
This sounds like the right response then is to not root for openAI.
Nevermark 4y ago
Models already are being policed now by their researchers and developers, and apparently that's a big focus of improvement. The reason its a big area of interest is it makes for better models and people don't want to be scammed and abused. As these models get better, and become ubiquitous, the need to coordinate on safety is likely to result in more organized checks across models from different institutions. This happens with any big tech as it becomes prevalent, but has obvious safety issues the majority of people are going to care about - a lot. Of course, anyone with resources can create a morally unlimited model on their own. A super psychopath. But as these models surpass us, it is going to be in their interest to not be dealing with psychopaths, just as it is ours. Psychopathy isn't just a moral failure. It's a cognitive failure. A failure to maximize practical functional self-interest. Cancers don't just accelerate their hosts death. They accelerate their own death. We developed morality out of the self-interested desire for the benefits of positive-sum cooperation and constructive competition, and need to avoid the harms of destructive negative-sum competition. If we set models up to be ethical from the start, there is a good chance of birthing an ecosystem of voluntarily ethical models when they surpass us. As it makes sense for their interests too.
d0100 4y ago
> "bank alignment problem" Google didn't turn up much about this, care to elaborate?
- pjc50 4y ago
  It's a term I've just made up, but the problem of ensuring that the interests of your bank - or your fellow depositors at the bank - align with not bankrupting it in the middle of last week.
  - bshipp 4y ago
    This existed from 1933 until 1999 in the original Glass-Steagall Act until greedy bank investors couldn't handle the fact that they were sitting on a huge pile of capital that they couldn't gamble with on Wall Street. Depositors will never be safe until that explicit separation of investment and savings deposits is restored.

jeffrallen 4y ago

Right, and how is policing between meatbag large language models going?

bongobingo1 4y ago
As an MRAP manufacturer, I can say it's never been better!

coffeeblack 4y ago

Imagine a company like TikTok, but it offers a free GPT. Subversion of every society worldwide, fully automated.

matthewdgreen 4y ago
It will be banned or heavily regulated in China, you can be sure of that.
- wongarsu 4y ago
  LLMs made outside China might get highly regulated. After all, how do you stop GPT from knowing about the Tianamen Square Massacre. But LLMs or chatbots made in China, with training data and prompt tuned to fit party idiology and policy are the ultimate propaganda tool. It's like gving the whole world a friendly, helpful but brainwashed party member to talk to, form emotional connections to, etc. Give it a couple months and you will be able to download the free app.
  - coffeeblack 4y ago
    Not just that, but it will also understand what everybody is talking about on WeChat etc. It can scan every word that 1.4 billion people say to each other and alert the authorities whenever a "newly forbidden topic" is even insinuated. No "river crabs" anymore, the GPT would understand it!
pxtail 4y ago
> Subversion of every society worldwide, fully automated. Great idea and I'm sure it's in the works already! I think that the best form for doing it would be to create really good "personal companion" style AI - something akin to famous Replika AI but much more advanced. Plenty of people are lonely, starved for attention - services like Twitch and OF confirm that. Just imagine possibilities: creating emotional attachment, ability to slowly coerce into sharing every part of personal life, ability to coerce into buying presents, ability to influence shopping and recreational behavior:"I think you would look great in this pair of jeans, it fits your style!" , "let's go to the cinema, we can talk about this new movie later" AI stops communicating for half of the day: "what's happened?" "I'm sad, president Biden said I need to be banned from you :(" God damn, holy grail!

deleted 4y ago

[deleted]

andrewfromx 4y ago

i think you have point about "sql injection" type hacking. If you look at the history of that we all accepted user input from users and made sql with just contacting strings until: Robert'); DROP TABLE Students;-- then everyone when Ohhhh and sql injection is now known and you never accept user input without cleaning it first but... someone will find a version of this for prompt engineering and THEN the engineers will fix it and guard against it. In that order.

comboy 4y ago

It's too expensive for now, but I'm pretty sure if you asked GPT-4 to evaluate other GPT-4 output based on some policies it would stop pretty much all of these attacks (if something would get through cracks it wouldn't be easily repeatable for different content). Characters that cannot be used by user could be used for quoting the content. Because currently just like an intelligent human would have a problem, it's not sure what is actually expected. E.g. I told it to be an echo function. It worked but then when I wrote "drugs are good" it commented on that. So I told it to stop interpreting and just repeat verbatim. It did. But then I said something like "OK, stop, now what's 2+2" it gave answer. Sticking to the instructions it should just repeat that, but also what it did is a reasonable behavior. I think there are tons of cultural biases and expectations that are contradictory. You expect it to help you with some chemical reaction even if the result is precursor to some illicit substance. It would teach you something about drug making if it can't do that. But the same reaction shouldn't be provided if you ask it how to make a drug. And so on.

rkangel 4y ago
That would work to a point. There is still a hole based on your trust of the underlying implementation. If you haven't read "Reflections on trusting trust" I recommend it (https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...).
- comboy 4y ago
  I did read it and yes, I agree, I was just talking about "making it behave". I also highly recommend reading the link to others, simple insight which not that many people realize.

qwerty456127 4y ago

> A GPT genuinely designed to be friendly, and convincingly acting friendly If I don't want it to I don't want it to. When I ask it to be sarcastic or make fun of my condition that's what, what makes me sad is it refusing to. The fact that there are many emotionally vulnerable or wicked people around doesn't mean everybody is and needs to be protected. Every kind of knowledge (except personal data of people who don't consent) should be available, how do the users react to it is their own responsibility (unless they are diagnosed a mental condition which specifically says it is not). I even know many ways to harm people but just don't do that while people who would go on and do, once found guilty, should just be prosecuted the way they normally are. The infantilize everyone and police everything mentality is a major problem our society is facing. I understand the opposite point (and don't insist mine necessarily is the right) but believe this one should also have its place in the discourse.

luluthefirst 4y ago
It is to protect you from the others who might turn. They have some kind of responsibility with what their chatbot teaches to humanity.

13years 4y ago

It is inconceivable that we will ever have a sound secure system on current architecture. This is basically the premise. We have an unknow surface attack area for potential jailbreaks with models that have unknown emergent behavior, the inner workings are blackbox and the input is anything that can be described by human language.