7 ms·

Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelli

Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes". The board was likely spooked by the recent breakthroughs (which were most likely achieved by combining transformers with another approach), and hit the panic button. Anything capable of "superhuman persuasion", especially prior to an election cycle, has tremendous consequences in the wrong hands.

PaulDavisThe1st 3y ago
Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion. So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.
- jjeaff 3y ago
  When you say persuasion, are you referring to fact based, logical argument? Because there are lots of other types of persuasion and certainly some work very well. Lying and telling people what they want to hear without too many details while dog whistling in ways that confirm their prejudices seems to be working pretty well for some people.
  - hnthrowaway0315 3y ago
    Just to add that a lot of people don't care about facts. In fact, if acting according to facts make me lose $$ I'd probably start building lies.
  - PaulDavisThe1st 3y ago
    * **that confirm their prejudices** * (my emphasis)
- hnthrowaway0315 3y ago
  Or, let's say, you don't need a lot of persuasion to guide an election. I mean we already have X, FB, and an army of bots.
- naasking 3y ago
  Even if it were true that human persuasion is of limited use in shifting opinions, the parent posted is talking about superhuman persuasion. I don't think we should just assume those are equally effective.
  - somenameforme 3y ago
    Do you think any rhetoric could ever persuade you to you adopt the opposite general worldview of what you currently have? I'm positive that it could not for me. The reason for this is not because I'm obstinate, but because my worldview is not formed on persuasion, but on lived experience. And I think this is true for the overwhelming majority of people. It's why our views tend to change as we age, and experience more of the world. You can even see this geographically. The reason many in South Texas might have a negative view of immigration while those in San Francisco might have a positive view of immigration is not because of persuasion differences, but because both places are strongly impacted by immigration but in very different ways. And this experience is what people associate with immigration in general, and so it forms people's worldview.
    - placebo 3y ago
      While I agree that human persuasion would probably not change a worldview built on lived experience, you can't know in advance what might be possible with superhuman persuasion. You might be led to believe that your experience was interpreted incorrectly, that things are different now or that you live in an illusion and don't even know who you are. There is no way to tell what the limits of psychological manipulation are for reprogramming your beliefs unless you are totally above any human doubt about everything, which is in itself a sad state to be in. I hope that persuaded you :)
      - somenameforme 3y ago
        Well, but I'm sure you'd accept that there are limits. Where we may differ is where those limits begin and where they end. In the end LLMs are not magical. All it's going to be able to do is present words to you. And how we respond to words is something that we can control. It's not like some series of words is just going to be able to completely reprogram you. Like here I expect there is 0% chance, even if I had a superhuman LLM writing words for me, that I could ever convince you that LLMs will not be able to convince you to hold any arbitrary position. It's because you've formed your opinion, it's not falsifiable, and so there's not a whole heck of a lot else to be done except have some fun debates like this where, if anything, we tend to work to strengthen our own opinions by finding and repairing any holes in them.
- hackerlight 3y ago
  > Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion. The Republican Party's base became isolationist and protectionist during 2015 and 2016 because their dear leader persuaded them.
  - RandomLensman 3y ago
    I don't think that aligns with the reality of the opinion formation. There was a strong subset of isolationist and protectionist views before 2015.
  - TMWNN 3y ago
    I know that by "dear leader" you mean to imply that Trump did something unfair/wrong/sinister/etc ("just like Hitler", amirite fellas?)., but a leader of a large group of people, by definition, is good at persuasion. Franklin Roosevelt moved the Democratic Party in a direction very different from its first century. The party's two previous presidential nominees were a Wall Street corporate lawyer (John W. Davis) and Al Smith who, despite also being a New York City resident and state governor, so opposed FDR by the end of his first term that he founded an influential anti-New Deal organization. During the Roosevelt years the Democrats lost significant support from traditional backers, but more than made up for it with gains elsewhere in what became the New Deal coalition. Similarly, under Trump the GOP lost support in wealthy suburbs but gained support elsewhere, such as Rust Belt states, Latinos (including places like South Florida and the Texas border region), blacks, and (according to current polls) young voters. We'll see whether one compensates for the other.
  - zztop44 3y ago
    I think it’s not clear that the causation flowed that way. I think it’s at least partially true that the Republican base was much more isolationist and protectionist than its “establishment” elite, so any significant candidate that played into that was going to get some level of support. That, combined with Donald Trump’s massive pre-existing celebrity, talent for showmanship, and utter shamelessness got him across the line. I think it’s fair to say that at least partially, Trump didn’t shift the base - rather he revealed that the base wasn’t where the establishment thought it was.
thepasswordis 3y ago
But they didn’t hit the panic button. They said Sam lied to them about something and fired him.
- adastra22 3y ago
  According to this article Sam has been telling the board that this new advance is not AGI and not anything to worry about (so they can keep selling it to MSFT), then the researchers involved went behind Sam's back and reported to the board directly, claiming that they'd created something that could-maybe-be AGI and it needs to be locked down. That's the claim at least.
  - deleted 3y ago
    [deleted]
  - sroussey 3y ago
    If that research team is unwanted at OpenAI, I know places they can go with coworkers writing to their boss’s boss.
    - sroussey 3y ago
      I meant to have the word not in there. Kinda changes the meaning
- deleted 3y ago
  [deleted]
ronhews 3y ago
Understandably, the board may be concerned about the potential consequences of AI-powered superhuman persuasion, particularly during an election cycle. Combining transformers with other approaches has led to recent breakthroughs, which could have significant implications.
sampo 3y ago
> Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes". Superhuman persuasion is Sam's area of expertise, so he would make that a priority when building chatbots.
somenameforme 3y ago
It seems much more likely that this was just referring to the ongoing situation with LLMs being able to create exceptionally compelling responses to questions that are completely and entirely hallucinated. It's already gotten to the point that I simply no longer use LLMs to learn about topics I am not already extremely familiar with, simply because hallucinations end up being such a huge time waster. Persuasion without accuracy is probably more dangerous to their business model than the world, because people learn extremely quickly not to use the models for anything you care about being right on.
- pbourke 3y ago
  Sounds like we need an AI complement to the Gell-Mann Amnesia effect.
Exoristos 3y ago
Which party is "the wrong hands"?
- Liquix 3y ago
  Any party with sufficient resources and motive to influence the outcome of an election. Outside of election season, this tech would be very dangerous in the hands of anyone seeking to influence the public for their own gain.
- bakuninsbart 3y ago
  Both? Parties in a democracy aren't supposed to be shepherds of the stupid masses, I know manipulation and misinformation is par for the course on both sides of the aisle, but that's a huge problem. Without informed, capable citizens, democracy dies a slow death.
- latexr 3y ago
  The original commenter didn’t mention a party. Please don’t polarise the discussion into a flame war. Whatever system exists won’t be used by “a party” all at once, but by individuals. Any of those, with any political affiliation, can be “the wrong hands”. I’ll offer a simple definition. The role of government is to serve the greater good of all people, thus the wrong hands are the ones which serve themselves or their own group above all.
meheleventyone 3y ago
Looking at humanity, persuasion seems to be an extremely low bar! Also for a superhuman trait is it that it’s capable of persuading anyone anything or rather that it’s able to persuade everyone about something. Power vs. Reach.
- 93po 3y ago
  I agree with this. Corporate news is complete and total obvious bullshit, but it overwhelmingly informs how people think about most anything.
column 3y ago
"especially prior to an election cycle" It looks like you are referring to the USA elections. 1. humanity != USA 2. USA are in a constant election cycle 3. there are always elections coming around the world, so it's never a good time
alkonaut 3y ago
I agree with this conclusion and it's also why I'm not that afraid of the AGI threat to the human race. AGI won't end the human race if "superhuman persuation" or "deception-as-a-service" does it first.
- nopromisessir 3y ago
  I feel this could be used in positive ways. Superhuman pursuation to do good stuff. That'll be a weird convo what is 'good'.
__MatrixMan__ 3y ago
We've built the web into a giant Skinner box. I find the claim dubious, but this is the sort of thing we ought to find at the forefront of our technology. It's where we've been going for a long time now.

PaulDavisThe1st 3y ago

Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion. So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.

jjeaff 3y ago
When you say persuasion, are you referring to fact based, logical argument? Because there are lots of other types of persuasion and certainly some work very well. Lying and telling people what they want to hear without too many details while dog whistling in ways that confirm their prejudices seems to be working pretty well for some people.
- hnthrowaway0315 3y ago
  Just to add that a lot of people don't care about facts. In fact, if acting according to facts make me lose $$ I'd probably start building lies.
- PaulDavisThe1st 3y ago
  * **that confirm their prejudices** * (my emphasis)
hnthrowaway0315 3y ago
Or, let's say, you don't need a lot of persuasion to guide an election. I mean we already have X, FB, and an army of bots.
naasking 3y ago
Even if it were true that human persuasion is of limited use in shifting opinions, the parent posted is talking about superhuman persuasion. I don't think we should just assume those are equally effective.
- somenameforme 3y ago
  Do you think any rhetoric could ever persuade you to you adopt the opposite general worldview of what you currently have? I'm positive that it could not for me. The reason for this is not because I'm obstinate, but because my worldview is not formed on persuasion, but on lived experience. And I think this is true for the overwhelming majority of people. It's why our views tend to change as we age, and experience more of the world. You can even see this geographically. The reason many in South Texas might have a negative view of immigration while those in San Francisco might have a positive view of immigration is not because of persuasion differences, but because both places are strongly impacted by immigration but in very different ways. And this experience is what people associate with immigration in general, and so it forms people's worldview.
  - placebo 3y ago
    While I agree that human persuasion would probably not change a worldview built on lived experience, you can't know in advance what might be possible with superhuman persuasion. You might be led to believe that your experience was interpreted incorrectly, that things are different now or that you live in an illusion and don't even know who you are. There is no way to tell what the limits of psychological manipulation are for reprogramming your beliefs unless you are totally above any human doubt about everything, which is in itself a sad state to be in. I hope that persuaded you :)
    - somenameforme 3y ago
      Well, but I'm sure you'd accept that there are limits. Where we may differ is where those limits begin and where they end. In the end LLMs are not magical. All it's going to be able to do is present words to you. And how we respond to words is something that we can control. It's not like some series of words is just going to be able to completely reprogram you. Like here I expect there is 0% chance, even if I had a superhuman LLM writing words for me, that I could ever convince you that LLMs will not be able to convince you to hold any arbitrary position. It's because you've formed your opinion, it's not falsifiable, and so there's not a whole heck of a lot else to be done except have some fun debates like this where, if anything, we tend to work to strengthen our own opinions by finding and repairing any holes in them.
hackerlight 3y ago
> Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion. The Republican Party's base became isolationist and protectionist during 2015 and 2016 because their dear leader persuaded them.
- RandomLensman 3y ago
  I don't think that aligns with the reality of the opinion formation. There was a strong subset of isolationist and protectionist views before 2015.
- TMWNN 3y ago
  I know that by "dear leader" you mean to imply that Trump did something unfair/wrong/sinister/etc ("just like Hitler", amirite fellas?)., but a leader of a large group of people, by definition, is good at persuasion. Franklin Roosevelt moved the Democratic Party in a direction very different from its first century. The party's two previous presidential nominees were a Wall Street corporate lawyer (John W. Davis) and Al Smith who, despite also being a New York City resident and state governor, so opposed FDR by the end of his first term that he founded an influential anti-New Deal organization. During the Roosevelt years the Democrats lost significant support from traditional backers, but more than made up for it with gains elsewhere in what became the New Deal coalition. Similarly, under Trump the GOP lost support in wealthy suburbs but gained support elsewhere, such as Rust Belt states, Latinos (including places like South Florida and the Texas border region), blacks, and (according to current polls) young voters. We'll see whether one compensates for the other.
- zztop44 3y ago
  I think it’s not clear that the causation flowed that way. I think it’s at least partially true that the Republican base was much more isolationist and protectionist than its “establishment” elite, so any significant candidate that played into that was going to get some level of support. That, combined with Donald Trump’s massive pre-existing celebrity, talent for showmanship, and utter shamelessness got him across the line. I think it’s fair to say that at least partially, Trump didn’t shift the base - rather he revealed that the base wasn’t where the establishment thought it was.

thepasswordis 3y ago

But they didn’t hit the panic button. They said Sam lied to them about something and fired him.

adastra22 3y ago
According to this article Sam has been telling the board that this new advance is not AGI and not anything to worry about (so they can keep selling it to MSFT), then the researchers involved went behind Sam's back and reported to the board directly, claiming that they'd created something that could-maybe-be AGI and it needs to be locked down. That's the claim at least.
- deleted 3y ago
  [deleted]
- sroussey 3y ago
  If that research team is unwanted at OpenAI, I know places they can go with coworkers writing to their boss’s boss.
  - sroussey 3y ago
    I meant to have the word not in there. Kinda changes the meaning
deleted 3y ago
[deleted]

ronhews 3y ago

Understandably, the board may be concerned about the potential consequences of AI-powered superhuman persuasion, particularly during an election cycle. Combining transformers with other approaches has led to recent breakthroughs, which could have significant implications.

sampo 3y ago

> Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes". Superhuman persuasion is Sam's area of expertise, so he would make that a priority when building chatbots.

somenameforme 3y ago

It seems much more likely that this was just referring to the ongoing situation with LLMs being able to create exceptionally compelling responses to questions that are completely and entirely hallucinated. It's already gotten to the point that I simply no longer use LLMs to learn about topics I am not already extremely familiar with, simply because hallucinations end up being such a huge time waster. Persuasion without accuracy is probably more dangerous to their business model than the world, because people learn extremely quickly not to use the models for anything you care about being right on.

pbourke 3y ago
Sounds like we need an AI complement to the Gell-Mann Amnesia effect.

Exoristos 3y ago

Which party is "the wrong hands"?

Liquix 3y ago
Any party with sufficient resources and motive to influence the outcome of an election. Outside of election season, this tech would be very dangerous in the hands of anyone seeking to influence the public for their own gain.
bakuninsbart 3y ago
Both? Parties in a democracy aren't supposed to be shepherds of the stupid masses, I know manipulation and misinformation is par for the course on both sides of the aisle, but that's a huge problem. Without informed, capable citizens, democracy dies a slow death.
latexr 3y ago
The original commenter didn’t mention a party. Please don’t polarise the discussion into a flame war. Whatever system exists won’t be used by “a party” all at once, but by individuals. Any of those, with any political affiliation, can be “the wrong hands”. I’ll offer a simple definition. The role of government is to serve the greater good of all people, thus the wrong hands are the ones which serve themselves or their own group above all.

meheleventyone 3y ago

Looking at humanity, persuasion seems to be an extremely low bar! Also for a superhuman trait is it that it’s capable of persuading anyone anything or rather that it’s able to persuade everyone about something. Power vs. Reach.

93po 3y ago
I agree with this. Corporate news is complete and total obvious bullshit, but it overwhelmingly informs how people think about most anything.

column 3y ago

"especially prior to an election cycle" It looks like you are referring to the USA elections. 1. humanity != USA 2. USA are in a constant election cycle 3. there are always elections coming around the world, so it's never a good time

alkonaut 3y ago

I agree with this conclusion and it's also why I'm not that afraid of the AGI threat to the human race. AGI won't end the human race if "superhuman persuation" or "deception-as-a-service" does it first.

nopromisessir 3y ago
I feel this could be used in positive ways. Superhuman pursuation to do good stuff. That'll be a weird convo what is 'good'.

__MatrixMan__ 3y ago

We've built the web into a giant Skinner box. I find the claim dubious, but this is the sort of thing we ought to find at the forefront of our technology. It's where we've been going for a long time now.