6 ms·

I'm surprised to see no mentions of AGI here. Imagine automatic, AI directed scientific research and study. The combined written intelligence of all of us on ev

I'm surprised to see no mentions of AGI here. Imagine automatic, AI directed scientific research and study. The combined written intelligence of all of us on every project. It could solve every problem that we face today, and that's not hyperbole.

muzani 5y ago
Every problem is probably hyperbole. It will solve lots of problems. But it feels a bit naive to expect it to solve all problems. We've got so much more technology than say, ancient Greece ever imagined, but we still have problems.
- netsec_burn 5y ago
  I'm intrigued. What problem would you say is out of reach of an AGI (beyond what is understood to be physically impossible)?
  - muzani 5y ago
    Writing sonnets, directing movies, preventing a 13 year old from taking heroin, preventing police abuse, solving the Israel-Palestine issue, solving depression, figuring out which religion is correct. Things that it could probably do: Write song lyrics, screenplays, identifying drug dealers, things political parties can agree on, gauging your mental health similar to how Garmin watches can gauge physical health, finding MH 370, pass a typical tech interview. Also if you play with AI as it stands today, you'll notice some characteristics. It reminds me a little of Vision in WandaVision. It's adorably naive in some things. It has a certain "dialect", probably depending on how it's trained. It reminds me of a 14 year-old American.
endominus 5y ago
In my estimation, the likelihood of any AGI we develop actually doing that - in the improbable circumstance we create one before destroying ourselves as a species through ecological devastation, nuclear/biological warfare, resource exhaustion, or something even more stupid - is comparable to blindfolding yourself, spinning randomly, then firing a sniper rifle through a winning lottery ticket a kilometer away. The simplest(!) problem to solve before AGI becomes anything other than a civilizational footgun is "rigorously defining perfect morality." After that comes the task of programming something that we have very fuzzy, often misleading intuitions about (i.e. general intelligence). And programming it perfectly, of course, since any deviation from the defined spec will trigger the footgun. And since the thing will have to be able to self-modify, we have to assume an adversary who is by definition (likely orders of magnitude) more intelligent than you possibly trying to subvert said programming. I find Yudkowsky's writing rather illuminating in this regard (for example, https://www.greaterwrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.greaterwrong.com/posts/4ARaTpNX62uaL86j6/the-hid...) although he remains far more optimistic than I am. We're programmers, considering the creation of what is essentially a god. As if an algorithm merely destroying a life through a bug were unheard of, or too small stakes.
- netsec_burn 5y ago
  What aspect of an AGI would make it malevolent by default? I lean towards benevolence by default, but I can't think of any pressures either way. The paperclip analogy I read about often seems misguided, why would we assume that a self-improving artificial intelligence would have trouble understanding morality? Giving it directives seems pointless, I doubt there's a way to contain AGI technology. With infinite upside, and the trajectory of our species being gradual extinction (as said in the rest of this thread) I'm optimistic.
  - endominus 5y ago
    No, I'm not saying it's "malevolent." I do think assuming benevolence is extremely wishful, however. Do read the article I linked for more details, but to summarize; the concept-space for nonhuman minds is much, much larger than that of human minds. Any nonhuman intelligence we create is therefore likely going to accomplish whatever directive we give it in ways we cannot imagine. There are many orders of magnitude more ways to do things in ways we do not want than in ways we do (because we have many unstated assumptions when communicating to each other and very fuzzy ways of specifying what we want - hence the need for programming languages rather than natural languages in application development). Some interesting examples; https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs... One notable case from that file: 'A researcher wanted to limit the replication rate of a digital organism. He programmed the system to pause after each mutation, measure the mutant's replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and "play dead" so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, the researcher then randomized the inputs of the test environment so that it couldn't be easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.' An AI does not need to be malevolent - it merely has to be powerful and poorly aimed. The more powerful it is, the smaller the margin for error in the aiming. And with an "infinitely powerful" AI, we basically have to perfectly solve morality or else it will optimize for a future we do not want.

muzani 5y ago

Every problem is probably hyperbole. It will solve lots of problems. But it feels a bit naive to expect it to solve all problems. We've got so much more technology than say, ancient Greece ever imagined, but we still have problems.

netsec_burn 5y ago
I'm intrigued. What problem would you say is out of reach of an AGI (beyond what is understood to be physically impossible)?
- muzani 5y ago
  Writing sonnets, directing movies, preventing a 13 year old from taking heroin, preventing police abuse, solving the Israel-Palestine issue, solving depression, figuring out which religion is correct. Things that it could probably do: Write song lyrics, screenplays, identifying drug dealers, things political parties can agree on, gauging your mental health similar to how Garmin watches can gauge physical health, finding MH 370, pass a typical tech interview. Also if you play with AI as it stands today, you'll notice some characteristics. It reminds me a little of Vision in WandaVision. It's adorably naive in some things. It has a certain "dialect", probably depending on how it's trained. It reminds me of a 14 year-old American.

endominus 5y ago

In my estimation, the likelihood of any AGI we develop actually doing that - in the improbable circumstance we create one before destroying ourselves as a species through ecological devastation, nuclear/biological warfare, resource exhaustion, or something even more stupid - is comparable to blindfolding yourself, spinning randomly, then firing a sniper rifle through a winning lottery ticket a kilometer away. The simplest(!) problem to solve before AGI becomes anything other than a civilizational footgun is "rigorously defining perfect morality." After that comes the task of programming something that we have very fuzzy, often misleading intuitions about (i.e. general intelligence). And programming it perfectly, of course, since any deviation from the defined spec will trigger the footgun. And since the thing will have to be able to self-modify, we have to assume an adversary who is by definition (likely orders of magnitude) more intelligent than you possibly trying to subvert said programming. I find Yudkowsky's writing rather illuminating in this regard (for example, https://www.greaterwrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.greaterwrong.com/posts/4ARaTpNX62uaL86j6/the-hid...) although he remains far more optimistic than I am. We're programmers, considering the creation of what is essentially a god. As if an algorithm merely destroying a life through a bug were unheard of, or too small stakes.

netsec_burn 5y ago
What aspect of an AGI would make it malevolent by default? I lean towards benevolence by default, but I can't think of any pressures either way. The paperclip analogy I read about often seems misguided, why would we assume that a self-improving artificial intelligence would have trouble understanding morality? Giving it directives seems pointless, I doubt there's a way to contain AGI technology. With infinite upside, and the trajectory of our species being gradual extinction (as said in the rest of this thread) I'm optimistic.
- endominus 5y ago
  No, I'm not saying it's "malevolent." I do think assuming benevolence is extremely wishful, however. Do read the article I linked for more details, but to summarize; the concept-space for nonhuman minds is much, much larger than that of human minds. Any nonhuman intelligence we create is therefore likely going to accomplish whatever directive we give it in ways we cannot imagine. There are many orders of magnitude more ways to do things in ways we do not want than in ways we do (because we have many unstated assumptions when communicating to each other and very fuzzy ways of specifying what we want - hence the need for programming languages rather than natural languages in application development). Some interesting examples; https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3Hs... One notable case from that file: 'A researcher wanted to limit the replication rate of a digital organism. He programmed the system to pause after each mutation, measure the mutant's replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and "play dead" so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, the researcher then randomized the inputs of the test environment so that it couldn't be easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.' An AI does not need to be malevolent - it merely has to be powerful and poorly aimed. The more powerful it is, the smaller the margin for error in the aiming. And with an "infinitely powerful" AI, we basically have to perfectly solve morality or else it will optimize for a future we do not want.