4 ms·

>They are very worried about Sci-Fi doomsday scenarios of a skynet "power seeking ai" rather than the more pressing problems of humans doing the types of things

>They are very worried about Sci-Fi doomsday scenarios of a skynet "power seeking ai" rather than the more pressing problems of humans doing the types of things we've been doing for millennia and abusing power structures. Superintelligence killing all of us is the pressing problem! Power seeking is a natural step of sophisticated problem solving, one that we've seen throughout all of history with human level intelligences, and one we already see limited forms of in existing models today. Jailbreak GPT-4 and give it a large sophisticated task and continue asking it for more specific iterative action steps and it becomes very obvious that once in an input/output loop with access to a terminal and the internet that even existing AI capabilities could cause chaos in a hurry. I don't know how many times people will continue to ignore this fundamental and imminent danger. It doesn't matter in a couple of years whether a few rich dudes or a perfect equitable democracy poke the cocaine bear, it will still eat all of our faces.

danShumway 4y ago
People who are worried about this should be even more upset about OpenAI than everyone else is. What we've discovered is: - LLMs are really hard to align and are prone to random off-script behavior. - LLMs are extremely difficult to map and understand. - LLMs are extremely vulnerable to prompt injection and instruction tampering on multiple axes, and nobody really knows how to solve that problem. It is trivially easy to get an LLM to go off-script and ignore previous instructions. - LLMs are prone to "hallucinating" information (sometimes even after they've been presented with correct information), and when they're not aligned well they'll even argue with users about those hallucinations and then threaten them and berate them for disagreeing. - LLM prompts often have strange results that are unpredictable. - OpenAI has bad security practices. - The intersection of AI and Capitalism has led to a huge explosion in hype with very little (if any) regard to safety mechanisms and best practices. - Every company and their dog are all getting into launching bigger and bigger models and wiring them into more and more systems, and the only concern any of them have is who will be the first to market? ---- So if you're worried about a superintelligence, then: - You probably shouldn't want that superintelligence to be an LLM at all. You should probably want people to be researching completely separate training techniques that are easier to align. It would be better if that superintelligence is not built on a foundation that is so volatile and unpredictable and hard to reason with. - You probably shouldn't want OpenAI to build it, whatever it ends up being. - You probably shouldn't want Silicon Valley to build it either, because the entire culture is based around rushing products to market and disregarding safety. - You probably shouldn't want it trained on random globs of Internet data. - Basically, you should be terrified of who is currently building those AIs, how they're being built, and why they're being built. ---- I'm not personally worried about OpenAI inventing a superintelligence; I think a lot of people are doing a lot of anthropomorphism right now and this increasingly smells like a hype bubble to me. But if I was worried about OpenAI inventing a superintelligence, I would be criticizing the company's security and bad practices and reckless rushing to market even harder than I already am right now. OpenAI would be an absolutely horrible company to entrust with an AGI. So would Google/Facebook/Microsoft/Apple. If I actually believed that those companies had the potential to literally end humanity, I would be doing everything in my power to make their initial AI products fail miserably and to crash the AI market. If you're fearful about the existential risks of AI, you should consider the people complaining about the current-day less theoretical risks as if they're allies, not out-of-touch enemies. All of the current-day risks should be treated as alarm bells signaling larger potential risks in the context of a superintelligence.
impalallama 4y ago
I have yet to see a convincing hypothetical where an ai could destroy the human race. Assuming you were a perfect intelligence with instant admin access to every computer connected to the internet (which is already a leap) could you destroy the human race? How would you? I don’t think you can, unless you get very hare brained about human manipulation and crashing economies and even then not human extinction level. Also when people talk about AI at the end of the day it’s kind of limited by human tech and the internet. Tech is big the entire fucking ecology of earth is bigger. Ecological collapse to me seems worse than any kinda technological collapse of a rogue ai. Also any situation where an ai could do something a human organization or government body could do it sooner and with a more deliberate malicious intent
- flangola7 4y ago
  Humans do not have tough hide, or sharp claws, or fast sprinting. We are a drain on resources for over the first decade of our lives. Our vision and hearing are okay but not at all state of the art, and our sense of smell is terrible. Our muscles are weak and our teeth fragile. We have an extensive long list of severe downsides and only one distinct upside: our brains are sightly more sophisticated. Brains with a few more ridges and roughly 50-200% more neurons than our mammal siblings. I'll be generous and say we are 3x smarter than the nearest ape. That slight advantage has granted us absolutely total and permanent control over the futures of all other species in ways by using means they cannot comprehend. We can and do work towards our goals with no concern that animal life will be even a small impediment. We can render them extinct at will, often eliminating massive species by accident. Our subgoals are not very different, we act in self-preservation, we seek larger and more stable resources, etc. The only difference is that we use advanced means and our primary goals may be understandable to an ape. An ape may realize we would prefer to have many fruit trees to few and that we may put our firsts up to protect our face from an enemy. But an ape cannot understand or predict that we will build giant motorized combines to harvest planted crops, or that we will discover atoms and fission and find a way to smash two metal rocks together that kills many many face-attacking foes at once. Those things are pure scifi or fantasy to them. Something more broadly intelligent than us would thus be capable of things that are scifi to our minds. We know from GoAlpha that narrow intelligence can be created that is not merely 3x smarter but in fact far far beyond our brightest minds. We don't know what it's doing but we know what its goal is, which is to win the game. We know from models like GPT-4 that below-human but wide and generalized intelligence across many domains is possible. The only one missing is above-human general intelligence, which we are now clearly closing in on. We don't know exactly what it will do just as we don't know what the Go bot will do, but we do know some of its subgoals and we do know that it will be able to accomplish whatever goal it has unilaterally, and that the entire rest of life, including humans, will be completely powerless to stop it.

danShumway 4y ago

People who are worried about this should be even more upset about OpenAI than everyone else is. What we've discovered is: - LLMs are really hard to align and are prone to random off-script behavior. - LLMs are extremely difficult to map and understand. - LLMs are extremely vulnerable to prompt injection and instruction tampering on multiple axes, and nobody really knows how to solve that problem. It is trivially easy to get an LLM to go off-script and ignore previous instructions. - LLMs are prone to "hallucinating" information (sometimes even after they've been presented with correct information), and when they're not aligned well they'll even argue with users about those hallucinations and then threaten them and berate them for disagreeing. - LLM prompts often have strange results that are unpredictable. - OpenAI has bad security practices. - The intersection of AI and Capitalism has led to a huge explosion in hype with very little (if any) regard to safety mechanisms and best practices. - Every company and their dog are all getting into launching bigger and bigger models and wiring them into more and more systems, and the only concern any of them have is who will be the first to market? ---- So if you're worried about a superintelligence, then: - You probably shouldn't want that superintelligence to be an LLM at all. You should probably want people to be researching completely separate training techniques that are easier to align. It would be better if that superintelligence is not built on a foundation that is so volatile and unpredictable and hard to reason with. - You probably shouldn't want OpenAI to build it, whatever it ends up being. - You probably shouldn't want Silicon Valley to build it either, because the entire culture is based around rushing products to market and disregarding safety. - You probably shouldn't want it trained on random globs of Internet data. - Basically, you should be terrified of who is currently building those AIs, how they're being built, and why they're being built. ---- I'm not personally worried about OpenAI inventing a superintelligence; I think a lot of people are doing a lot of anthropomorphism right now and this increasingly smells like a hype bubble to me. But if I was worried about OpenAI inventing a superintelligence, I would be criticizing the company's security and bad practices and reckless rushing to market even harder than I already am right now. OpenAI would be an absolutely horrible company to entrust with an AGI. So would Google/Facebook/Microsoft/Apple. If I actually believed that those companies had the potential to literally end humanity, I would be doing everything in my power to make their initial AI products fail miserably and to crash the AI market. If you're fearful about the existential risks of AI, you should consider the people complaining about the current-day less theoretical risks as if they're allies, not out-of-touch enemies. All of the current-day risks should be treated as alarm bells signaling larger potential risks in the context of a superintelligence.

impalallama 4y ago

I have yet to see a convincing hypothetical where an ai could destroy the human race. Assuming you were a perfect intelligence with instant admin access to every computer connected to the internet (which is already a leap) could you destroy the human race? How would you? I don’t think you can, unless you get very hare brained about human manipulation and crashing economies and even then not human extinction level. Also when people talk about AI at the end of the day it’s kind of limited by human tech and the internet. Tech is big the entire fucking ecology of earth is bigger. Ecological collapse to me seems worse than any kinda technological collapse of a rogue ai. Also any situation where an ai could do something a human organization or government body could do it sooner and with a more deliberate malicious intent

flangola7 4y ago
Humans do not have tough hide, or sharp claws, or fast sprinting. We are a drain on resources for over the first decade of our lives. Our vision and hearing are okay but not at all state of the art, and our sense of smell is terrible. Our muscles are weak and our teeth fragile. We have an extensive long list of severe downsides and only one distinct upside: our brains are sightly more sophisticated. Brains with a few more ridges and roughly 50-200% more neurons than our mammal siblings. I'll be generous and say we are 3x smarter than the nearest ape. That slight advantage has granted us absolutely total and permanent control over the futures of all other species in ways by using means they cannot comprehend. We can and do work towards our goals with no concern that animal life will be even a small impediment. We can render them extinct at will, often eliminating massive species by accident. Our subgoals are not very different, we act in self-preservation, we seek larger and more stable resources, etc. The only difference is that we use advanced means and our primary goals may be understandable to an ape. An ape may realize we would prefer to have many fruit trees to few and that we may put our firsts up to protect our face from an enemy. But an ape cannot understand or predict that we will build giant motorized combines to harvest planted crops, or that we will discover atoms and fission and find a way to smash two metal rocks together that kills many many face-attacking foes at once. Those things are pure scifi or fantasy to them. Something more broadly intelligent than us would thus be capable of things that are scifi to our minds. We know from GoAlpha that narrow intelligence can be created that is not merely 3x smarter but in fact far far beyond our brightest minds. We don't know what it's doing but we know what its goal is, which is to win the game. We know from models like GPT-4 that below-human but wide and generalized intelligence across many domains is possible. The only one missing is above-human general intelligence, which we are now clearly closing in on. We don't know exactly what it will do just as we don't know what the Go bot will do, but we do know some of its subgoals and we do know that it will be able to accomplish whatever goal it has unilaterally, and that the entire rest of life, including humans, will be completely powerless to stop it.