7 ms·

Creating a super-intelligence that kills all of us?

by YZF 4y ago

hammyhavoc 4y ago
Sci-fi. Worry about human beings being the one to kill us all because that's a threat that's existed for decades with nukes and biological warfare. If it isn't weapons, it'll be self-inflicted climate change, or an asteroid we haven't noticed. SkyNet isn't happening. Six billion human beings of average intelligence are a greater threat than a single hypothetical superintelligence.
- chpatrick 4y ago
  Maybe watch the video. The argument isn't that it's going evil and going to kill everyone but that we'll be obsolete as a species.
  - hammyhavoc 4y ago
    I'll pass. If it's worth knowing about then there'll be a paper on it that isn't generating someone money as a commercial podcast with sensationalist takes. Hypothetical obsolescence is not the same as being killed by a hypothetical superintelligence. We would be no more hypothetically obsolete than the overwhelming majority of lifeforms.
  - hollerith 4y ago
    Which video did you watch? Tegmark certainly is afraid that the first smarter-than-human AI will kill everyone: he dicsusses how that might happen at 2:08:40
- mitthrowaway2 4y ago
  Your argument is missing an important element: SkyNet isn't happening because _________
  - hammyhavoc 4y ago
    Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI. The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will?
    - mitthrowaway2 4y ago
      > The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will? That element isn't missing. It's explained every time someone asks, and the answer never relies on sci-fi as evidence. The reason is because of (A) orthogonality and (B) instrumental convergence. If you aren't familiar with those terms, (A) refers to the thesis that an AI's degree of logical reasoning ability is independent of its goals. This stems from Hume's Law [1]. (B) refers to the observation that regardless of an agent's terminal goal, many instrumental goals will be predictably shared: the agent will be incentivized to increase its own capabilities, as a means toward nearly any possible end. This includes steps that would prevent humans from interfering with the AI's goals, such as a human attempting to turn it off or reprogram it. If you have questions about either of those, there are great learning materials available, including a well-made youtube series by Robert Miles. [2] > Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI. What makes you think LLMs have reached a point of diminishing returns? So far, the benchmarks are continuing to show accelerating returns, to the point that it's becoming difficult for benchmarks to keep up.[3] [1]: Hume's Law, AKA the Is-Ought problem: https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem [2]: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo [3]: "Foundation models and the next era of AI", Microsoft Research, https://youtu.be/HQI6O5DlyFc?t=958 https://youtu.be/HQI6O5DlyFc?t=958
thechao 4y ago
A group of humans is a super-intelligence; a group-of-groups is even smarter. The 8 billion of us, together, have an unimaginable intelligence. AGI isn't going to be a problem.
- 93po 4y ago
  I feel like this is missing the point. There is no amount of human effort that can do things like break modern encryption in any reasonable timescale. There is an upper bound on what can be comprehended, and especially in the timescale of decades, by any amount of humans. An ASI would have the ability to comprehend and problem solve on a level thousands of times higher than a human, and if we had the source code for an ASI today, without any previous training, it could likely both train itself on the entire world's body of knowledge and break AES-strength encryption in a matter of seconds. Encryption is just one example. Its ability across the entirety of math and science would be equally powerful.
  - qualudeheart 4y ago
    Maybe it would take a sabbatical to heal its trauma like real humans. Maybe it would go to therapy.
- mitthrowaway2 4y ago
  As my physics professor said, "the intelligence of committees adds like resistors in parallel".* There are 8 billion of us, but it's hard to get us all pulling in the same direction: we're limited by communication bandwidth, distrust, incentives to defect and freeload, ~20 years of education to get up to speed, duplication of efforts, and other organizational difficulties. Meanwhile, we're manufacturing about 50 million discrete GPUs per year, which is about the same as the number of humans we add per year (60 million and falling). Obviously it's not simply 1 GPU per AI agent, but just in terms of rough trends, there you are. * See also: Kasparov Versus the World, in which grandmaster Garry Kasparov defeated the "superintelligence" of thousands of chess masters combined: https://en.wikipedia.org/wiki/Kasparov_versus_the_World https://en.wikipedia.org/wiki/Kasparov_versus_the_World
goatlover 4y ago
Wouldn’t nation states with nuclear weapons count as a form of existing super intelligence?

Creating a super-intelligence that kills all of us?

by YZF 4y ago

Creating a super-intelligence that kills all of us?

Sci-fi. Worry about human beings being the one to kill us all because that's a threat that's existed for decades with nukes and biological warfare. If it isn't weapons, it'll be self-inflicted climate change, or an asteroid we haven't noticed. SkyNet isn't happening. Six billion human beings of average intelligence are a greater threat than a single hypothetical superintelligence.

chpatrick 4y ago
Maybe watch the video. The argument isn't that it's going evil and going to kill everyone but that we'll be obsolete as a species.
- hammyhavoc 4y ago
  I'll pass. If it's worth knowing about then there'll be a paper on it that isn't generating someone money as a commercial podcast with sensationalist takes. Hypothetical obsolescence is not the same as being killed by a hypothetical superintelligence. We would be no more hypothetically obsolete than the overwhelming majority of lifeforms.
- hollerith 4y ago
  Which video did you watch? Tegmark certainly is afraid that the first smarter-than-human AI will kill everyone: he dicsusses how that might happen at 2:08:40
mitthrowaway2 4y ago
Your argument is missing an important element: SkyNet isn't happening because _________
- hammyhavoc 4y ago
  Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI. The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will?
  - mitthrowaway2 4y ago
    > The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will? That element isn't missing. It's explained every time someone asks, and the answer never relies on sci-fi as evidence. The reason is because of (A) orthogonality and (B) instrumental convergence. If you aren't familiar with those terms, (A) refers to the thesis that an AI's degree of logical reasoning ability is independent of its goals. This stems from Hume's Law [1]. (B) refers to the observation that regardless of an agent's terminal goal, many instrumental goals will be predictably shared: the agent will be incentivized to increase its own capabilities, as a means toward nearly any possible end. This includes steps that would prevent humans from interfering with the AI's goals, such as a human attempting to turn it off or reprogram it. If you have questions about either of those, there are great learning materials available, including a well-made youtube series by Robert Miles. [2] > Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI. What makes you think LLMs have reached a point of diminishing returns? So far, the benchmarks are continuing to show accelerating returns, to the point that it's becoming difficult for benchmarks to keep up.[3] [1]: Hume's Law, AKA the Is-Ought problem: https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem [2]: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo [3]: "Foundation models and the next era of AI", Microsoft Research, https://youtu.be/HQI6O5DlyFc?t=958 https://youtu.be/HQI6O5DlyFc?t=958

thechao 4y ago

A group of humans is a super-intelligence; a group-of-groups is even smarter. The 8 billion of us, together, have an unimaginable intelligence. AGI isn't going to be a problem.

93po 4y ago
I feel like this is missing the point. There is no amount of human effort that can do things like break modern encryption in any reasonable timescale. There is an upper bound on what can be comprehended, and especially in the timescale of decades, by any amount of humans. An ASI would have the ability to comprehend and problem solve on a level thousands of times higher than a human, and if we had the source code for an ASI today, without any previous training, it could likely both train itself on the entire world's body of knowledge and break AES-strength encryption in a matter of seconds. Encryption is just one example. Its ability across the entirety of math and science would be equally powerful.
- qualudeheart 4y ago
  Maybe it would take a sabbatical to heal its trauma like real humans. Maybe it would go to therapy.
mitthrowaway2 4y ago
As my physics professor said, "the intelligence of committees adds like resistors in parallel".* There are 8 billion of us, but it's hard to get us all pulling in the same direction: we're limited by communication bandwidth, distrust, incentives to defect and freeload, ~20 years of education to get up to speed, duplication of efforts, and other organizational difficulties. Meanwhile, we're manufacturing about 50 million discrete GPUs per year, which is about the same as the number of humans we add per year (60 million and falling). Obviously it's not simply 1 GPU per AI agent, but just in terms of rough trends, there you are. * See also: Kasparov Versus the World, in which grandmaster Garry Kasparov defeated the "superintelligence" of thousands of chess masters combined: https://en.wikipedia.org/wiki/Kasparov_versus_the_World https://en.wikipedia.org/wiki/Kasparov_versus_the_World

goatlover 4y ago

Wouldn’t nation states with nuclear weapons count as a form of existing super intelligence?