4 ms·

Has Toner (or someone with like-minded views) filled in the blanks between "GPT-4" and "Terminator Judgement Day" in a believable way? I've read what Yudkowsky

Has Toner (or someone with like-minded views) filled in the blanks between "GPT-4" and "Terminator Judgement Day" in a believable way? I've read what Yudkowsky writes but it all sounds so fantastical that it's, at least to me, more like an episode of The Twilight Zone or The X-Files than something resembling reality. Is Toner really in the "nuke the datacenters" camp? If so, was her placement on the board not a mistake from the beginning?

dmix 3y ago
I have a feeling EA[1] is like anarcho capitalist communities (or just inverse for socialism) where you have a good idea like markets==good then apply it like a hammer to everything you see in life. Any hint of gov/statism is treated as the enemy, regardless of its IRL negative impact on markets or net freedom, or whatever positive end goal they envision. OpenAI is (hypothetically) progressing to AGI, therefore bad. Whether GPT specifically is a half a foot on a 1000km journey or potentially insignificant, it doesn't matter to them as they've pre-selected any potential for progress as all bad. [1] if you search "Helen Toner" on YouTube it's just a collection of videos of her talking at EA conferences. So she's a believer
- deleted 3y ago
  [deleted]
ah765 3y ago
I think the middle ground where several of the OpenAI board members were trying for is to "responsibly develop AGI", which means developing at a moderate pace while trying to avoid kicking off an investment gold rush through making heavily commercial use cases, and spending a substantial amount of resources on promising safety research (such as Ilya's work). In my opinion, it was not a very strong position because the allure of money and trying to be the biggest is too strong (as we're seeing now), but I think it was at least coherent.
- remarkEon 3y ago
  So, no, the blanks have not been filled in then. Because, for those in Toner's camp, that middle ground is just "progress from GPT to T-1000, but slowly", right? Yudkowsky talks about AIs 3D printing themselves into biological entities and killing us all because humans are an inefficient use of matter. It's not a strong position to me because it sounds ridiculous, not because there's greed involved.
- dmix 3y ago
  It takes some level of delusion of grandeur to think half a board of a single non profit that just happens to be the first mover can stop the full forces of American capitalism - although I kind of respect the drive/purpose. They could easily lose any power they had to guide the industry, it was a huge gamble. I remember reading a Harvard business school study showing the first mover advantage repeatedly turned out to be ineffective in the tech industry as there is a looong series of early winners dying out to later market entrants like Friendster->FB, Google, a bunch of dotcom era Ecommerce companies predating Amazon, etc. They need full industry/society buy in - at an ideological level - to win this battle, they won't win through backroom dealing in a boardroom while losing 90% of their own staff.
Sol- 3y ago
No idea if the concerns are realistic, but dismissing them because they are fantastical surely is a fallacy, no? We currently almost have scifi-level AI we can talk to on our smartphones, which would have been fantastical only a few years ago.
- adastra22 3y ago
  The concerns are unrealistic because there are massive unusual and unvalidated assumptions in multiple levels, and lynchpin arguments that depend on utterly unsupported assumptions.
cjbprime 3y ago
I don't know whether I have "like-minded views", but I think the risk is there, and it is not fantastical to me because the AGI doesn't need to "do" anything outside of humans assisting it -- no killer robots or nanobots. It just needs to persuade humans to kill for it, or gather resources for it, and so on. And because it's an AGI, it has superhuman abilities for persuasion. The intuition for GPT-4 being on the way towards superhuman intelligence or persuasion abilities is that the only significant difference between it and GPT-3 appears to be the amount of compute power applied to training. The previous US President was much less than superhuman, and he still managed to seriously threaten democracy in the US, and by extension cause a minor existential risk to civilization. Imagine what a more intelligent agent with enough attention to be personally persuasive instead of generically persuasive could achieve for itself, given a goal. (If you think that LLMs can't express goals or preferences, read some of the Bing Sydney transcripts.)
- gmerc 3y ago
  Exactly, AGI is overrated because the world isn’t run by its smartest people, ChatGPT would make a better US or Russian president at this point compared with treasonous or genocidal geriatrics
adastra22 3y ago
No one has, no.
bakuninsbart 3y ago
In his recent Ted talk, Ilya talks about AI as 'brains in a computer', and explains why he believes they aren't that far off from AGI. [0] GPT4 is a substantially worse software engineer than I am, but it is 10x GPT3. And if we 10x for at most 2 or 3 iterations, it could be better than me. While it is unclear that we can get there, it doesn't seem implausible to me, and would have deep implications for human existence. That is already true if it just stayed at that level, but what most doomers are really worried about is the possibility of GPT8 being as good or a better AI researcher than most or all humans. [0] https://www.youtube.com/watch?v=SEkGLj0bwAU https://www.youtube.com/watch?v=SEkGLj0bwAU
gizmo385 3y ago
I feel like the “Terminator Judgement Day” scenario is a strawman more often than not. There are reasons to push for the safe and responsible development of AI that aren’t based on the belief that AI is literally going to launch the nukes. The biggest risks I see from AI over the next 2 decades are: 1. A complete devolution in what is true (because creating convincing fakes will be trivial). From politics to the way that people interact with each other on a daily basis, this seems like it has the potential to sew division 2. the societal turmoil/upheaval arising from the mass unemployment (and subsequent unemployability) of millions of people without the social safety nets in place to handle that. When people are suddenly unable to work and support their families and those who have consolidated the power have no interest in sharing the wealth, there will inevitably be pushback.
FeepingCreature 3y ago
> fill in the blanks What exactly are you looking for here, proof of concept? This stuff is inherently speculative - because if it wasn't speculative, we'd probably be dead.

dmix 3y ago

I have a feeling EA[1] is like anarcho capitalist communities (or just inverse for socialism) where you have a good idea like markets==good then apply it like a hammer to everything you see in life. Any hint of gov/statism is treated as the enemy, regardless of its IRL negative impact on markets or net freedom, or whatever positive end goal they envision. OpenAI is (hypothetically) progressing to AGI, therefore bad. Whether GPT specifically is a half a foot on a 1000km journey or potentially insignificant, it doesn't matter to them as they've pre-selected any potential for progress as all bad. [1] if you search "Helen Toner" on YouTube it's just a collection of videos of her talking at EA conferences. So she's a believer

deleted 3y ago
[deleted]

ah765 3y ago

I think the middle ground where several of the OpenAI board members were trying for is to "responsibly develop AGI", which means developing at a moderate pace while trying to avoid kicking off an investment gold rush through making heavily commercial use cases, and spending a substantial amount of resources on promising safety research (such as Ilya's work). In my opinion, it was not a very strong position because the allure of money and trying to be the biggest is too strong (as we're seeing now), but I think it was at least coherent.

remarkEon 3y ago
So, no, the blanks have not been filled in then. Because, for those in Toner's camp, that middle ground is just "progress from GPT to T-1000, but slowly", right? Yudkowsky talks about AIs 3D printing themselves into biological entities and killing us all because humans are an inefficient use of matter. It's not a strong position to me because it sounds ridiculous, not because there's greed involved.
dmix 3y ago
It takes some level of delusion of grandeur to think half a board of a single non profit that just happens to be the first mover can stop the full forces of American capitalism - although I kind of respect the drive/purpose. They could easily lose any power they had to guide the industry, it was a huge gamble. I remember reading a Harvard business school study showing the first mover advantage repeatedly turned out to be ineffective in the tech industry as there is a looong series of early winners dying out to later market entrants like Friendster->FB, Google, a bunch of dotcom era Ecommerce companies predating Amazon, etc. They need full industry/society buy in - at an ideological level - to win this battle, they won't win through backroom dealing in a boardroom while losing 90% of their own staff.

Sol- 3y ago

No idea if the concerns are realistic, but dismissing them because they are fantastical surely is a fallacy, no? We currently almost have scifi-level AI we can talk to on our smartphones, which would have been fantastical only a few years ago.

adastra22 3y ago
The concerns are unrealistic because there are massive unusual and unvalidated assumptions in multiple levels, and lynchpin arguments that depend on utterly unsupported assumptions.

cjbprime 3y ago

I don't know whether I have "like-minded views", but I think the risk is there, and it is not fantastical to me because the AGI doesn't need to "do" anything outside of humans assisting it -- no killer robots or nanobots. It just needs to persuade humans to kill for it, or gather resources for it, and so on. And because it's an AGI, it has superhuman abilities for persuasion. The intuition for GPT-4 being on the way towards superhuman intelligence or persuasion abilities is that the only significant difference between it and GPT-3 appears to be the amount of compute power applied to training. The previous US President was much less than superhuman, and he still managed to seriously threaten democracy in the US, and by extension cause a minor existential risk to civilization. Imagine what a more intelligent agent with enough attention to be personally persuasive instead of generically persuasive could achieve for itself, given a goal. (If you think that LLMs can't express goals or preferences, read some of the Bing Sydney transcripts.)

gmerc 3y ago
Exactly, AGI is overrated because the world isn’t run by its smartest people, ChatGPT would make a better US or Russian president at this point compared with treasonous or genocidal geriatrics

adastra22 3y ago

No one has, no.

bakuninsbart 3y ago

In his recent Ted talk, Ilya talks about AI as 'brains in a computer', and explains why he believes they aren't that far off from AGI. [0] GPT4 is a substantially worse software engineer than I am, but it is 10x GPT3. And if we 10x for at most 2 or 3 iterations, it could be better than me. While it is unclear that we can get there, it doesn't seem implausible to me, and would have deep implications for human existence. That is already true if it just stayed at that level, but what most doomers are really worried about is the possibility of GPT8 being as good or a better AI researcher than most or all humans. [0] https://www.youtube.com/watch?v=SEkGLj0bwAU https://www.youtube.com/watch?v=SEkGLj0bwAU

gizmo385 3y ago

I feel like the “Terminator Judgement Day” scenario is a strawman more often than not. There are reasons to push for the safe and responsible development of AI that aren’t based on the belief that AI is literally going to launch the nukes. The biggest risks I see from AI over the next 2 decades are: 1. A complete devolution in what is true (because creating convincing fakes will be trivial). From politics to the way that people interact with each other on a daily basis, this seems like it has the potential to sew division 2. the societal turmoil/upheaval arising from the mass unemployment (and subsequent unemployability) of millions of people without the social safety nets in place to handle that. When people are suddenly unable to work and support their families and those who have consolidated the power have no interest in sharing the wealth, there will inevitably be pushback.

FeepingCreature 3y ago

> fill in the blanks What exactly are you looking for here, proof of concept? This stuff is inherently speculative - because if it wasn't speculative, we'd probably be dead.