38 ms·

Both those 1%s are very unlikely to be anywhere close to being true. The sci-fi version of AGI that people are imaging now will never exist. No one has any ide

by jmull 4y ago

Both those 1%s are very unlikely to be anywhere close to being true. The sci-fi version of AGI that people are imaging now will never exist. No one has any idea how to create AGI now, and therefore, no one knows what it might actually look like if it ever does. Any “solutions” people come up with now will have nothing to do with the problems a real AGI might pose. If people want to get together and talk about this stuff, that’s great. But if someone wants you to give money to fund research into this, you’re being scammed.

medstrom 4y ago
There's nothing substantive in this post, just unexplained opinions...
- jmull 4y ago
  The main substantive point I made is: "No one has any idea how to create AGI now" Correct, no? The rest follows by basic logic. You can't craft solutions to problems you don't know anything about. You can't assign probabilities to outcomes without any information on the inputs or mechanisms of the system.
  - medstrom 4y ago
    Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? 2. You can always assign probabilities. I'll give you an example: say I'm a scary dictator and I ask you: "is A or B correct?" You have no idea what I mean by A or B, so you'd assign 50-50 if you had to bet something dear on the outcome of my totally unfair context-free question. Perhaps you happen to know of research showing that people more often set A as the correct answer in this sort of game, so you might assign 60-40, let's say. That's a probability estimate!! Same principle for AI. You don't know much, but what you do know can still be used for a judgment. Supposing you do not believe in souls, then brains are just computers, then it's possible in principle for a computer to become sentient and intelligent, then it's just a matter of time 'til they do, whether it happens 10 years from now or 1000. Maybe you feel the human race will go extinct shortly, and that'll be another valid factor in your estimate of how much this is to worry about. But either you accept that it CAN happen and discuss how many precautions to take today, or you're understandably unconcerned because it appears prima facie impossible to you. I imagine that for you to be unconcerned, you'd have to have judged that a being cannot create a being more intelligent than itself. (If you're using a different reasoning, please enlighten me) But if that's your reasoning... what did evolution do? It has no brain, yet it created an intelligent being. And if you carve out a special clause for evolution-like processes, humans could still just simulate an accelerated evolution in a lab/computer and wait for a superintelligence to pop out. Does that not make sense?
    - polygamous_bat 4y ago
      I will only respond to 1. > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? Wrong. Here's the logical chain: 1. For someone to do X, they will need to know how to do X. (Unless you think it's all just alchemy, and we're boiling urine to distill gold.) 2. For creation of X to be imminent, someone's knowledge of how to create X will have to be almost complete. 3. Currently, no one knows how to do X. 4. Thus, creation of X is nowhere near imminent.
    - jmull 4y ago
      > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? I think it does lead to that. How will people create something they have no idea how to create... Perhaps it will somehow emerge accidentally, but that's very unlikely. I tried to make this clear before, but I'll state it clearly: I think discussing is completely fine. We all have our hobbies. It's convincing people there's a real problem and taking real money to solve a fake problem that I object to. That's a con. If AGI does happen and they start doing things we GIs don't like, the problem is going to be around specifically what they are doing, why they are doing it and what we don't like about it. The solutions are going to be in terms of those specific problem spaces. There would also be a feedback loop of response, counter-response, which is fundamentally unpredictable. > evolution-like processes Given a planet-sized environment and 4-5 billion years, I think we can say emergent GI is possible. But that doesn't help someone with a lab. I think we'd do better by studying how brains work, really. I think we're probably going to do best by starting with simpler brains and see if we can trace the path evolution has take (thus saving ourself billions of years of random trial and error). > That's a probability estimate!! Let me wrap up on this: I estimate a 99.99% likelihood that people discussing the problems of AGI now will fail to correctly predict the nature of AGI and their resultant problems, and therefore will waste any and all effort/money spent now in that direction. Someone actually seriously interested in this who wants to put resources to it now would want to put money into a long-term trust, to be used once the problem is clear. However, that assumes the existence of a stable trustee mechanism that remains uncorrupted over the long term to ensure the money is used as intended. I'm not how that is accomplished, but some long-term foundations seem to have done OK. (E.g., the Nobel Foundation doesn't do too bad. The Peace Prize is all politics, and all of the prizes have some politics, but overall it's mostly on-track.) If there's a 10% chance such a trust function well enough long enough for 50% of the money to be spent in search of solutions to the actual problems, that's 5% effectiveness. Far better than the 0.01% effectiveness I estimate for spending the money now. Note: the ancestor post of this discussion also estimated a 0.01% effectivness: > If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Still, I think an EA could easily come up with another use for the same resources that does a lot better than 5%, so we're back to not spending any resources on this now. Much better to have future EAs work on it, once the problems are clear.
tim333 4y ago
Well, opinions differ. I'd guess the probability of there being AGI as in computers able to think as well as a human in most ways by 2300 as close to 100%. Things we have now are getting closer eg Deepmind's Gato and GPT-3.
- polygamous_bat 4y ago
  > Things we have now are getting closer eg Deepmind's Gato and GPT-3. For someone who actually works in the field, let me inform you that you've been had by the marketing folks. Deepmind's Gato is as close to AGI as connecting a line of cars with chains and calling it a train. Gato is glorified if else statement over multiple models, and not really "general" at all.
- danaris 4y ago
  Absolutely none of the recent interesting machine-learning successes are even going in the right direction to ever become AGI. There is as much similarity between GPT-3 and an AGI as there is between a human and one of those novelty fortune-telling machines with racist caricatures and names like Zardoz.
jasonwatkinspdx 4y ago
Exactly. Researching AI safety today is like researching how to reduce the proliferation of black hole bombs.
- medstrom 4y ago
  What would it make sensible to start researching the proliferation of black hole bombs, for you?

by jmull 4y ago

medstrom 4y ago

There's nothing substantive in this post, just unexplained opinions...

jmull 4y ago
The main substantive point I made is: "No one has any idea how to create AGI now" Correct, no? The rest follows by basic logic. You can't craft solutions to problems you don't know anything about. You can't assign probabilities to outcomes without any information on the inputs or mechanisms of the system.
- medstrom 4y ago
  Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? 2. You can always assign probabilities. I'll give you an example: say I'm a scary dictator and I ask you: "is A or B correct?" You have no idea what I mean by A or B, so you'd assign 50-50 if you had to bet something dear on the outcome of my totally unfair context-free question. Perhaps you happen to know of research showing that people more often set A as the correct answer in this sort of game, so you might assign 60-40, let's say. That's a probability estimate!! Same principle for AI. You don't know much, but what you do know can still be used for a judgment. Supposing you do not believe in souls, then brains are just computers, then it's possible in principle for a computer to become sentient and intelligent, then it's just a matter of time 'til they do, whether it happens 10 years from now or 1000. Maybe you feel the human race will go extinct shortly, and that'll be another valid factor in your estimate of how much this is to worry about. But either you accept that it CAN happen and discuss how many precautions to take today, or you're understandably unconcerned because it appears prima facie impossible to you. I imagine that for you to be unconcerned, you'd have to have judged that a being cannot create a being more intelligent than itself. (If you're using a different reasoning, please enlighten me) But if that's your reasoning... what did evolution do? It has no brain, yet it created an intelligent being. And if you carve out a special clause for evolution-like processes, humans could still just simulate an accelerated evolution in a lab/computer and wait for a superintelligence to pop out. Does that not make sense?
  - polygamous_bat 4y ago
    I will only respond to 1. > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? Wrong. Here's the logical chain: 1. For someone to do X, they will need to know how to do X. (Unless you think it's all just alchemy, and we're boiling urine to distill gold.) 2. For creation of X to be imminent, someone's knowledge of how to create X will have to be almost complete. 3. Currently, no one knows how to do X. 4. Thus, creation of X is nowhere near imminent.
  - jmull 4y ago
    > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? I think it does lead to that. How will people create something they have no idea how to create... Perhaps it will somehow emerge accidentally, but that's very unlikely. I tried to make this clear before, but I'll state it clearly: I think discussing is completely fine. We all have our hobbies. It's convincing people there's a real problem and taking real money to solve a fake problem that I object to. That's a con. If AGI does happen and they start doing things we GIs don't like, the problem is going to be around specifically what they are doing, why they are doing it and what we don't like about it. The solutions are going to be in terms of those specific problem spaces. There would also be a feedback loop of response, counter-response, which is fundamentally unpredictable. > evolution-like processes Given a planet-sized environment and 4-5 billion years, I think we can say emergent GI is possible. But that doesn't help someone with a lab. I think we'd do better by studying how brains work, really. I think we're probably going to do best by starting with simpler brains and see if we can trace the path evolution has take (thus saving ourself billions of years of random trial and error). > That's a probability estimate!! Let me wrap up on this: I estimate a 99.99% likelihood that people discussing the problems of AGI now will fail to correctly predict the nature of AGI and their resultant problems, and therefore will waste any and all effort/money spent now in that direction. Someone actually seriously interested in this who wants to put resources to it now would want to put money into a long-term trust, to be used once the problem is clear. However, that assumes the existence of a stable trustee mechanism that remains uncorrupted over the long term to ensure the money is used as intended. I'm not how that is accomplished, but some long-term foundations seem to have done OK. (E.g., the Nobel Foundation doesn't do too bad. The Peace Prize is all politics, and all of the prizes have some politics, but overall it's mostly on-track.) If there's a 10% chance such a trust function well enough long enough for 50% of the money to be spent in search of solutions to the actual problems, that's 5% effectiveness. Far better than the 0.01% effectiveness I estimate for spending the money now. Note: the ancestor post of this discussion also estimated a 0.01% effectivness: > If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Still, I think an EA could easily come up with another use for the same resources that does a lot better than 5%, so we're back to not spending any resources on this now. Much better to have future EAs work on it, once the problems are clear.

tim333 4y ago

Well, opinions differ. I'd guess the probability of there being AGI as in computers able to think as well as a human in most ways by 2300 as close to 100%. Things we have now are getting closer eg Deepmind's Gato and GPT-3.

polygamous_bat 4y ago
> Things we have now are getting closer eg Deepmind's Gato and GPT-3. For someone who actually works in the field, let me inform you that you've been had by the marketing folks. Deepmind's Gato is as close to AGI as connecting a line of cars with chains and calling it a train. Gato is glorified if else statement over multiple models, and not really "general" at all.
danaris 4y ago
Absolutely none of the recent interesting machine-learning successes are even going in the right direction to ever become AGI. There is as much similarity between GPT-3 and an AGI as there is between a human and one of those novelty fortune-telling machines with racist caricatures and names like Zardoz.

jasonwatkinspdx 4y ago

Exactly. Researching AI safety today is like researching how to reduce the proliferation of black hole bombs.

medstrom 4y ago
What would it make sensible to start researching the proliferation of black hole bombs, for you?