3 ms·

Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely

Thank you for engaging with me. I have two questions. 1. "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? 2. You can always assign probabilities. I'll give you an example: say I'm a scary dictator and I ask you: "is A or B correct?" You have no idea what I mean by A or B, so you'd assign 50-50 if you had to bet something dear on the outcome of my totally unfair context-free question. Perhaps you happen to know of research showing that people more often set A as the correct answer in this sort of game, so you might assign 60-40, let's say. That's a probability estimate!! Same principle for AI. You don't know much, but what you do know can still be used for a judgment. Supposing you do not believe in souls, then brains are just computers, then it's possible in principle for a computer to become sentient and intelligent, then it's just a matter of time 'til they do, whether it happens 10 years from now or 1000. Maybe you feel the human race will go extinct shortly, and that'll be another valid factor in your estimate of how much this is to worry about. But either you accept that it CAN happen and discuss how many precautions to take today, or you're understandably unconcerned because it appears prima facie impossible to you. I imagine that for you to be unconcerned, you'd have to have judged that a being cannot create a being more intelligent than itself. (If you're using a different reasoning, please enlighten me) But if that's your reasoning... what did evolution do? It has no brain, yet it created an intelligent being. And if you carve out a special clause for evolution-like processes, humans could still just simulate an accelerated evolution in a lab/computer and wait for a superintelligence to pop out. Does that not make sense?

polygamous_bat 4y ago
I will only respond to 1. > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? Wrong. Here's the logical chain: 1. For someone to do X, they will need to know how to do X. (Unless you think it's all just alchemy, and we're boiling urine to distill gold.) 2. For creation of X to be imminent, someone's knowledge of how to create X will have to be almost complete. 3. Currently, no one knows how to do X. 4. Thus, creation of X is nowhere near imminent.
jmull 4y ago
> "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? I think it does lead to that. How will people create something they have no idea how to create... Perhaps it will somehow emerge accidentally, but that's very unlikely. I tried to make this clear before, but I'll state it clearly: I think discussing is completely fine. We all have our hobbies. It's convincing people there's a real problem and taking real money to solve a fake problem that I object to. That's a con. If AGI does happen and they start doing things we GIs don't like, the problem is going to be around specifically what they are doing, why they are doing it and what we don't like about it. The solutions are going to be in terms of those specific problem spaces. There would also be a feedback loop of response, counter-response, which is fundamentally unpredictable. > evolution-like processes Given a planet-sized environment and 4-5 billion years, I think we can say emergent GI is possible. But that doesn't help someone with a lab. I think we'd do better by studying how brains work, really. I think we're probably going to do best by starting with simpler brains and see if we can trace the path evolution has take (thus saving ourself billions of years of random trial and error). > That's a probability estimate!! Let me wrap up on this: I estimate a 99.99% likelihood that people discussing the problems of AGI now will fail to correctly predict the nature of AGI and their resultant problems, and therefore will waste any and all effort/money spent now in that direction. Someone actually seriously interested in this who wants to put resources to it now would want to put money into a long-term trust, to be used once the problem is clear. However, that assumes the existence of a stable trustee mechanism that remains uncorrupted over the long term to ensure the money is used as intended. I'm not how that is accomplished, but some long-term foundations seem to have done OK. (E.g., the Nobel Foundation doesn't do too bad. The Peace Prize is all politics, and all of the prizes have some politics, but overall it's mostly on-track.) If there's a 10% chance such a trust function well enough long enough for 50% of the money to be spent in search of solutions to the actual problems, that's 5% effectiveness. Far better than the 0.01% effectiveness I estimate for spending the money now. Note: the ancestor post of this discussion also estimated a 0.01% effectivness: > If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Still, I think an EA could easily come up with another use for the same resources that does a lot better than 5%, so we're back to not spending any resources on this now. Much better to have future EAs work on it, once the problems are clear.

polygamous_bat 4y ago

I will only respond to 1. > "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? Wrong. Here's the logical chain: 1. For someone to do X, they will need to know how to do X. (Unless you think it's all just alchemy, and we're boiling urine to distill gold.) 2. For creation of X to be imminent, someone's knowledge of how to create X will have to be almost complete. 3. Currently, no one knows how to do X. 4. Thus, creation of X is nowhere near imminent.

jmull 4y ago

> "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right? I think it does lead to that. How will people create something they have no idea how to create... Perhaps it will somehow emerge accidentally, but that's very unlikely. I tried to make this clear before, but I'll state it clearly: I think discussing is completely fine. We all have our hobbies. It's convincing people there's a real problem and taking real money to solve a fake problem that I object to. That's a con. If AGI does happen and they start doing things we GIs don't like, the problem is going to be around specifically what they are doing, why they are doing it and what we don't like about it. The solutions are going to be in terms of those specific problem spaces. There would also be a feedback loop of response, counter-response, which is fundamentally unpredictable. > evolution-like processes Given a planet-sized environment and 4-5 billion years, I think we can say emergent GI is possible. But that doesn't help someone with a lab. I think we'd do better by studying how brains work, really. I think we're probably going to do best by starting with simpler brains and see if we can trace the path evolution has take (thus saving ourself billions of years of random trial and error). > That's a probability estimate!! Let me wrap up on this: I estimate a 99.99% likelihood that people discussing the problems of AGI now will fail to correctly predict the nature of AGI and their resultant problems, and therefore will waste any and all effort/money spent now in that direction. Someone actually seriously interested in this who wants to put resources to it now would want to put money into a long-term trust, to be used once the problem is clear. However, that assumes the existence of a stable trustee mechanism that remains uncorrupted over the long term to ensure the money is used as intended. I'm not how that is accomplished, but some long-term foundations seem to have done OK. (E.g., the Nobel Foundation doesn't do too bad. The Peace Prize is all politics, and all of the prizes have some politics, but overall it's mostly on-track.) If there's a 10% chance such a trust function well enough long enough for 50% of the money to be spent in search of solutions to the actual problems, that's 5% effectiveness. Far better than the 0.01% effectiveness I estimate for spending the money now. Note: the ancestor post of this discussion also estimated a 0.01% effectivness: > If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent. Still, I think an EA could easily come up with another use for the same resources that does a lot better than 5%, so we're back to not spending any resources on this now. Much better to have future EAs work on it, once the problems are clear.