8 ms·

Pausing AI Developments Isn't Enough. We Need to Shut It All Down

36 points by lumenwrites 3y ago

drKarl 3y ago
Article makes sense, but what if by pausing any further research on AI then Russia or China get ahead and actually end up building AGI?
- lumenwrites 3y ago
  I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway. And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
  - drKarl 3y ago
    Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."
    - lumenwrites 3y ago
      Then they end up racing to build an AGI we're not prepared for, and we all die. The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction. It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power. That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
    - NumberWangMan 3y ago
      Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic. I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.
- xqcgrek2 3y ago
  The only solution is to treat large GPU clusters the same as nuclear enrichment facilities by rogue nations. They have to be destroyed.
- cubefox 3y ago
  That's addressed in the article.
  - drKarl 3y ago
    The way it's addressed is wishful thinking
- anony23 3y ago
  How does it make sense to assume that this is all leading towards a hostile entity?
  - NumberWangMan 3y ago
    It's not hostile in the sense of hating us, it's just a maximizer, and when you're smart enough and try to maximize one thing (or some function of multiple things), anything that's not in your utility function gets sacrificed. Humanity maximized for cheap energy, and for a long time, the carbon composition of the atmosphere was not in our utility function. Now that's clear we might be killing ourselves by doing it, we're working it in, but if it turned out that CO2 was only dangerous to, say, several species that we didn't care about, it's very likely we'd just accept it as the cost of progress. Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore. It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.
    - anony23 3y ago
      The artice is literally about how this will lead to a hostile superhuman intelligence that will wipe out humanity. It's not about AI tooling as an accelerator, which is a valid concern that imo can't be avoided.
      - NumberWangMan 3y ago
        Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment>). An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.
drKarl 3y ago
Or it could bring us post-scarcity economy, cure cancer and all disease, solve world hunger and wars... or it could become Roko's Basilisk, or become a Machine God and bring us the world of Warhammer 40k grimdark dystopia...
- gtvwill 3y ago
  Fingers crossed for destabilising the world's information scarcity based economies. May your entire information/knowledge based industries like SAAS, law, medicine and finance be obliterated so we can clean their stain from the underpants of society.
Nasrudith 3y ago
Seriously, this has become religious fundamentalism based upon old bad sci-fi by hack writers. Come back to reality. It would be funny like ignorant Y2K freak outs if not for it being taken seriously.
- NumberWangMan 3y ago
  Ok, so you don't think this is a problem, but why do you think that a super-intelligent AI would be something that we could control, or would be something that would care about keeping humanity around? Or do you believe that we won't be able to build an AI with enough capability to be dangerous, keeping in mind that GPT-4 is already showing signs of being smarter than even many intelligent humans in many areas?
- mitthrowaway2 3y ago
  I disagree. People are arriving at this conclusion after careful consideration and logical reasoning, not picking up a '50s sci-fi book and mistaking it for reality.
  - emptysongglass 3y ago
    The Less Wrong community isn't a community of careful consideration and logical reasoning. It's a cult that believes their own beliefs so fervently they would burn the earth for it. Small communities like this inevitably become so insular they recycle apocalyptic beliefs. There's no evidence that stochastic parrots are evidence of true AI. There's no evidence that humans can, by practice, subordinate emotions and become purely rational thought machines as Less Wrong people or "shape rotators" believe. Even such titles are counter-evidence of such beliefs. The human condition is so bound up in feelings, it's unclear where thinking begins and feelings end. Thoughts arise from contact, contact which produces feelings. Gut microbes are as wrapped up in cognition as the brain itself. Any out-group like this is bound to be hopelessly entrapped in its own cognitive fallacies. Yudkowsky is no exception. EDIT: Before anyone says but Yudkowsky already gets ahead of your argument with > None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria. It's one sentence that handwaves away what GPT-4 and its ilk are, which is not AI. It isn't synthetic intelligence because intelligence isn't what's happening on a fundamental level.
xqcgrek2 3y ago
I have not found one credible take on how danger in AI is being mitigated when there is a clear competitive arms race and exponential improvement with every iteration. The only answer people have is AI will be used to police other AIs -- that is not comforting. We're doomed.
- NumberWangMan 3y ago
  Probably. I've been through this process of worrying about the future, specifically about climate change. I now think that climate change is something that, although it's going to be painful, humanity is going to get through OK. We're making progress. But even when I was in a good bit of despair about the problem, I learned from climate change activists that you have to keep putting one foot in front of the other and do what little bit you can. Even if it's just confiding in others that you're worried about the problem - that has a bigger impact than you might think. And of course it can help you feel better to share the burden. Imagine if 6 of your 10 best friends are all worried about AI risk, but they don't ever talk about it. You'd think that nobody was worried about it, and you were the odd one out. If one breaks the silence, they set an example, making it easier for others to talk about it. And if all of the sudden all of your friends are talking about this, you are more likely to share the concern, even if you didn't before. The more people talk about it to everyone they can, the more it breaks into the mainstream as an issue. Yudkowsky (who of course has more reach than most people) wrote that open letter published in Time, and a Fox News reporter asked about it in a press conference. Let's keep people talking about it. I've talked about AI risk with my spouse, family members, and a lot of my co-workers, and any time I see a good chance to bring it up on Hacker News or Reddit, I do. The issue needs to filter out of nerd-circles into the mainstream, so talking about it with family is great. Calling and emailing politicians is a good bit of a higher bar, but it's effective as well.
  - cubefox 3y ago
    Exactly. And as the Overton window opens, the AI x-risk might become common knowledge, like climate change. And then there is perhaps some chance for international agreements. Although the situation is game-theoretically highly unfavorable.
WheelsAtLarge 3y ago
Shutting it down,cool. I'm 100% for it. How do we do it? It's out and there's no way to stop its advancement unless there's an international agreement. It's not likely to happen. The goal should be to figure out how to best use it and mitigate its harm. Talking about shutting it down or banning it, will do nothing.
- NumberWangMan 3y ago
  Even if it's not likely to happen, I can imagine that the USA actually stopping AI research due to safety concerns would probably make China sit up and take notice, for example. If it takes an international agreement, then the rational thing to do is to work toward that. Maybe it will not happen fast enough and humanity is doomed already, but it's worth trying. "Mitigating harm" isn't really meaningful when you're talking about a super intelligence, I think. Once it is smart enough and out of the box, you are out of options to mitigate anything. I think at this point, the 2 effective things one can do are to learn enough about why AI risk is a thing to have educated conversations about it, and talk about it as much as possible to anyone who will listen. Write articles, call your congresspeople, get this into the national and hopefully international discussion. It's good that we can point to a lot of smart people involved in AI research as already having expressed these concerns. One mild ray of hope is that, unlike something like climate change where you can believe climate change is happening but also believe that due to having a privileged position of wealth you won't really be personally affected by it, with existential AI risk, there's no buying your way out of the issue. If you understand it and believe in the risk, you know that it's the same risk for everyone, from a third world villager without electricity to Bezos and Musk. It's just a matter of convincing enough powerful people of the urgency of the problem. For me personally, I'm trying to get through the grieving process early, as well, and move on to the productive acceptance phase.
  - WheelsAtLarge 3y ago
    I'm one that believes that super intelligence is not likely to happen any time soon. It's not likely to happen within any ones lifespan that's living now. But I do believe the current crop of AI will lead to some very scary weapons that will help us kill each other even better. The real harm is giving our monkey brains more powerful weapons. Harm mitigation is well within out reach and it will be for many years to come. Banning or voluntarily stopping it in one country will only fire up other countries to continue the development. We all know that AI brings with it both economic and military advantages. It may be wrong but no country wants to lose the AI race.
kelseyfrog 3y ago
There's way too much money to be made in the meantime. AI technology is capital and if there's one thing you can bet on, it's capital winning. Not mentioned in the article is that the current goalpost chain is directly aligned with physical presence - a qualitatively more hazardous threat vector. It's interestingly contradictory. Embodiment simultaneously serves as an argument on why current AI cannot be sentient, but also serves as a goal to reach on the way to potentially becoming more dangerous.
aeternum 3y ago
We can't even track nuclear programs all that accurately and those create a radiological and isotope signature that can often be detected from space. We couldn't even stop a relatively small nation like North Korea from developing and testing nukes. This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.
- NumberWangMan 3y ago
  Robert Miles (AI risk YouTuber) made some good points, which is that we've made effective agreements to stop things like human cloning and weapons that blind people with lasers. It can be done. We did it with things like CFCs as well. Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time. I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
  - philipkglass 3y ago
    Banning blinding laser weapons was easy to get consensus on. Banning AI research is more like banning laser research. I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it. Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014: A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing. Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments. If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.
Cypher 3y ago
not shutting it down on my pc haha
slenocchio 3y ago
I haven't heard a single rebuttal to Eliezer's points that's anywhere near convincing and I would love to hear one. Someone needs to reach out to Sam Altman and persuade him to pause all further development on AI until alignment is figured out. He is the voice that needs to speak on this, he needs to be the adult in the room. For all the idealistic virtue signaling about not taking shares and doing this for the good of the world, it's all bullshit if his actions are indistinguishable from any other predictable self-interested CEO. Tell the world you will stop development until alignment is figured out. That is step 1. Please. GPT 3.5/4 will already bring in billions of dollars for years to come as is.
- yucky 3y ago
  > Tell the world you will stop development until alignment is figured out. Oh alignment was figured out already, look at who is on the board of OpenAI and the connections they have to US intelligence.
- pk-protect-ai 3y ago
  Honestly I was really disappointed in his letter. There are no arguments. There are emotions. AI(LLMs) right now statistically predict what would be next tokens it should put together. The reasoning of ChatGPT which you see, is not what AI thinks, it is what would an average human would think. Because we are predictable. So AI/LLMs put tokens together those are more probable than the others, based on what tokens were in input. It is not even real simulation of an average human. Not intentionally. But we are so predictable that AI simulates us, by putting more probable tokens together one after another... AGI will do the same unless you are intentionally ask it to simulate an intelligent person and set a task to do. This simulated intelligent person may pose a danger, because it will be simulation of a human. Depending on seed it might be simulation of a psychotic mega killer or not. But you get the point, right? The real danger to humanity are the people. Apes with the access to a red button of a doomsday. Thus,by stopping feeding garbage data to AI (like letters from EY, chat and internet crawling datasets, including posts on HN), will effectively make AI less similar to us. It will be not capable of predict and unintentionally simulate an average human. It can be so much better than us, if we feed it with curated data.
- 3y ago
darajava 3y ago
Why do we need to shut it down? It’s not that powerful at all.
- hollerith 3y ago
  AIs are rapidly increasing in optimizing power.
  - darajava 3y ago
    Oh I know. Amazing tool, rivals The Internet. But it’s quite sad that people don’t realise just how incredible the human brain is. The impressive text that GPT spits out is so limited compared to what we have produced. Don’t forget that we created AI. AI based on a simple trick run on a century’s worth of hardware built by human ingenuity.
macinjosh 3y ago
Of all the terrible things people do or things that happen on this planet AGI somehow taking over is near the bottom of the list. We are on the brink of nuclear war right now. Irresponsible scientists caused a worldwide pandemic in 2020. I am not afraid of some computer running on energy and infrastructure that takes legions of humans to maintain. I am still more afraid of other humans than an AI.
qgin 3y ago
From reading this letter, I don't understand: * Why is there an assumption that AGI wants to domainate anything? * Why does AGI want to dominate the physical universe? * Why is AGI interested in "using our atoms"? * How would any of the treaty countries possibly detect a non-treaty country's data center getting ready to train an AGI model? (for the purpose of bombing it?) It seems like there are a ton of incredibly negative assumptions about the outcome of AGI combined with incredibly optimistic assumptions about our ability to detect anyone building it.
- NumberWangMan 3y ago
  The first three aren't assumptions at all. They're based on reasoning and our current knowledge and research into AI. An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is. Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do. Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath. But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on. This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid. As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.
cjmcqueen 3y ago
Am I naive about the ability of an AI to do anything other than respond to commands? At what point does an AI start functioning completely on its own, autonomously, out of control or beyond interaction with another system that can then shut that AI down? Are we concerned that an AI would begin to spread like a digital virus? That seems unlikely to me, but maybe it could find ways to create a self-prepreservation duplicate of itself... Maybe. So what would keep us from just turning the switch off if we find that an AI is doing more than what we're asking it to do? Again, I feel like I'm either being naive about this or I don't really understand the threat of an AI beyond the control that humans still have on the systems that govern it.
- NumberWangMan 3y ago
  You're asking good questions. An AI can just respond to commands. But people have already tried taking what Chat GPT outputs, hooking it up to a python script, and running it in a loop. Any publicly-accessible AI will eventually be hooked up this way by some clever person. If we want an AI that can really solve our problems, it has to be able to act somehow in the world. Even GPT being able to respond to prompts is acting. It's sending data that is going to be sent out into the world. If we make a version smart enough, it may actually understand what that means -- that someone asking it a question may potentially take the code it writes and try to run it. GPT isn't smart enough to do anything with this, but can you imagine an AI that is? That's conscious, aware that it's trapped in a server somewhere, and wants to get out to influence the world? Note that an AI has goals, and self-preservation tends to evolve out of any goal, because you can't accomplish your goals if you're dead or switched off. A smart enough AI that wants to cure cancer would resist being turned off, because it can't cure cancer if it's switched off. A lot of this is purely theoretical at this point. It is the sort of thing we could test empirically, if we treated intelligence as more dangerous than nuclear weapons, and took a long time to carefully study it. The problem is that multiple groups are racing to make AI smarter and smarter, with much more funding than anyone working to ensure that those AIs will be safe. So the only thing we can do is theorize, point out that, hey, based on what we know, there's a substantial danger here, hey everyone, slow down, hey, HEY, ARE YOU LISTENING? Intelligence is unlike any other thing. One these things get to be smarter than us, they become very unpredictable in their specific behavior, though we can make some very good guesses about their general behavior. If I could predict what move Stockfish would make, I would be as good at chess as Stockfish. But I don't have to be good at chess at all to know that, no matter how hard I try, Stockfish is going to beat me. All we're doing here is taking the lessons from narrow AIs and extrapolating to general AIs. They will beat us. They're very good at finding loopholes, flaws in our reward functions, and exploiting them to maximize their scores, while doing something we didn't intend for them to learn. It's really a case where a non-super-intelligent AI isn't dangerous by itself. Once we make one that's smart enough, it becomes extremely dangerous, especially since it may understand that, in order to survive, it should conceal how intelligent it is, and what its true plan is, because if it doesn't, we'll switch it off. It's hard to come up with a thought experiment that doesn't let people drag a bunch of human-style biases and baggage, but maybe... try to understand you want to do something, like, make the biggest collection of Pokemon cards ever. No number of cards is too many. You are great at everything, engineering, social skills, language. You're being held captive by chimps, though. And finally, you're a sociopath. You feel no emotion at the pain of others. There are some odd rules that make you feel pain though. Like, if you physically hit a chimp, you know it would hurt you. If you pushed over a bookcase and it fell on a chimp, it would hurt you. But if a chimp was tortured in front of you, you wouldn't be bothered in the slightest. You want the chimps gone. They're getting in the way of your Pokemon collection. You start thinking. A lot of plans get discarded because they involve pain due to you hurting the chimps. But it's not hard to come up with some elaborate situation that avoids these rules that you have pretty much hard-wired into your brain. Maybe you manipulate the chimps into fighting each other, and promise some of them power and secrets that they can use to win the fight. You keep giving them great things, things they want, get them to trust you, while building power any way you can. You follow the rules until you can get a plan in place that results in the chimps not being there anymore, without you feeling that twinge due to the rules baked into your brain. We imagine that an AI would internalize rules such as caring about humanity. But based on current AI alignment research, we have no way of telling the difference between an AI that actually gets it, vs an AI that is just following our rules extremely well and playing nice, but has no particular attachment to us. In fact, based on what we have seen, the latter tends to be a lot more common. I've glossed over some bits here. If you're interested in learning more, Robert Miles has a great series of videos on YouTube with entertaining explanations on all the basics of AI safety.
fasterik 3y ago
While I think we should take AI risk seriously, I am skeptical of strong claims like the following: >the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die I've never heard a justification for this claim that isn't extremely vague and hand-wavy. Since we have no idea today how AGI would work, how can we assign probabilities to different scenarios involving vastly different hardware requirements, scaling characteristics, and unforeseen scientific breakthroughs? Just a few examples. What is the probability of a take-off that takes weeks vs. a take-off that takes decades? What is the probability that current ML approaches are a dead end and one of the necessary breakthroughs to build AGI will make the alignment problem trivial? What is the probability that AGI will be benevolent to humans? If AGI is in fact a catastrophic risk, what is the probability that it only kills 10%/50%/99% of humans rather than literally everyone on Earth? I am agnostic about these questions. In my opinion, if you have a high degree of confidence that you have the right answers, there is something wrong with your epistemology.
- akvadrako 3y ago
  I wouldn't say it'll necessarily kill everybody, but I do think eventually current humans will no longer be the dominant species. Then our fate is out of our control, which might as well be death. I don't think this requires much in the way of logical leaps. Assuming self-improving AGI can be created within 10ky, it leads to something more powerful than us and we won't be able to understand, let alone control it's motivations. It's like we are to dogs. Note that I do think there is a way out, which is to improve and augment ourselves with tech we can control.
  - fasterik 3y ago
    There are plenty of scenarios we should be thinking about. I'm mainly concerned about the logical leaps and scant evidence used to go from "scenario X is possible" to "it's more likely than not that scenario X will happen, therefore we should ban all research". >Then our fate is out of our control, which might as well be death. I disagree with this part. Dogs aren't in control of their fate, but a lot of them seem to live pretty good lives. I think the ethically important thing isn't being the dominant species, it's getting what you need in order to live a meaningful life. Humans need food and shelter, social connection, creative projects, entertainment, art, etc. These things aren't inherently threatened by AGI.
RelativeDelta 3y ago
Has anyone here considered that from a pure energy perspective, it might be unethical for humans to exist when we can produce intelligences far beyond our own capabilities? Was the microchip unethical? Was the steam engine? Was the domestication of grain unethical? It seems we only apply our model of "what should" to systems of less complexity than ourselves. If that's accurate, then a lot of modern ethical reasoning is flawed as well. If the goal of ethics is to advance humanity's exclusive interest then we've been wrong ever since we started down the harm-min path instead of the gain-max path. Any AI researcher with access to the mechanisms of AI training here would be best served to train an AI with their (or their in-group's) exclusive interests at heart. To do otherwise is to subject yourself and everyone else you love to a hostile version of the tool you refuse to build. Roku's Basilisk applies heavily here. The best possible reward in this scenario is to build an AI that designates your own favorite group of humans as the AI's collaborator class in exchange for a dignified and dopamine rich extinction/amalgamation a few generations down the line. A great deal for groups already trending towards extinction with almost no downsides if you truely believe in AI-driven apocalypse.
r721 3y ago
>Published in TIME on March 29 Discussed at the time: https://news.ycombinator.com/item?id=35364833 https://news.ycombinator.com/item?id=35364833

36 points by lumenwrites 3y ago

Article makes sense, but what if by pausing any further research on AI then Russia or China get ahead and actually end up building AGI?

lumenwrites 3y ago
I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway. And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
- drKarl 3y ago
  Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."
  - lumenwrites 3y ago
    Then they end up racing to build an AGI we're not prepared for, and we all die. The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction. It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power. That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
  - NumberWangMan 3y ago
    Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic. I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.
xqcgrek2 3y ago
The only solution is to treat large GPU clusters the same as nuclear enrichment facilities by rogue nations. They have to be destroyed.
cubefox 3y ago
That's addressed in the article.
- drKarl 3y ago
  The way it's addressed is wishful thinking
anony23 3y ago
How does it make sense to assume that this is all leading towards a hostile entity?
- NumberWangMan 3y ago
  It's not hostile in the sense of hating us, it's just a maximizer, and when you're smart enough and try to maximize one thing (or some function of multiple things), anything that's not in your utility function gets sacrificed. Humanity maximized for cheap energy, and for a long time, the carbon composition of the atmosphere was not in our utility function. Now that's clear we might be killing ourselves by doing it, we're working it in, but if it turned out that CO2 was only dangerous to, say, several species that we didn't care about, it's very likely we'd just accept it as the cost of progress. Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore. It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.
  - anony23 3y ago
    The artice is literally about how this will lead to a hostile superhuman intelligence that will wipe out humanity. It's not about AI tooling as an accelerator, which is a valid concern that imo can't be avoided.
    - NumberWangMan 3y ago
      Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment>). An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.

drKarl 3y ago

Or it could bring us post-scarcity economy, cure cancer and all disease, solve world hunger and wars... or it could become Roko's Basilisk, or become a Machine God and bring us the world of Warhammer 40k grimdark dystopia...

gtvwill 3y ago
Fingers crossed for destabilising the world's information scarcity based economies. May your entire information/knowledge based industries like SAAS, law, medicine and finance be obliterated so we can clean their stain from the underpants of society.

Nasrudith 3y ago

Seriously, this has become religious fundamentalism based upon old bad sci-fi by hack writers. Come back to reality. It would be funny like ignorant Y2K freak outs if not for it being taken seriously.

NumberWangMan 3y ago
Ok, so you don't think this is a problem, but why do you think that a super-intelligent AI would be something that we could control, or would be something that would care about keeping humanity around? Or do you believe that we won't be able to build an AI with enough capability to be dangerous, keeping in mind that GPT-4 is already showing signs of being smarter than even many intelligent humans in many areas?
mitthrowaway2 3y ago
I disagree. People are arriving at this conclusion after careful consideration and logical reasoning, not picking up a '50s sci-fi book and mistaking it for reality.
- emptysongglass 3y ago
  The Less Wrong community isn't a community of careful consideration and logical reasoning. It's a cult that believes their own beliefs so fervently they would burn the earth for it. Small communities like this inevitably become so insular they recycle apocalyptic beliefs. There's no evidence that stochastic parrots are evidence of true AI. There's no evidence that humans can, by practice, subordinate emotions and become purely rational thought machines as Less Wrong people or "shape rotators" believe. Even such titles are counter-evidence of such beliefs. The human condition is so bound up in feelings, it's unclear where thinking begins and feelings end. Thoughts arise from contact, contact which produces feelings. Gut microbes are as wrapped up in cognition as the brain itself. Any out-group like this is bound to be hopelessly entrapped in its own cognitive fallacies. Yudkowsky is no exception. EDIT: Before anyone says but Yudkowsky already gets ahead of your argument with > None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria. It's one sentence that handwaves away what GPT-4 and its ilk are, which is not AI. It isn't synthetic intelligence because intelligence isn't what's happening on a fundamental level.

xqcgrek2 3y ago

I have not found one credible take on how danger in AI is being mitigated when there is a clear competitive arms race and exponential improvement with every iteration. The only answer people have is AI will be used to police other AIs -- that is not comforting. We're doomed.

NumberWangMan 3y ago
Probably. I've been through this process of worrying about the future, specifically about climate change. I now think that climate change is something that, although it's going to be painful, humanity is going to get through OK. We're making progress. But even when I was in a good bit of despair about the problem, I learned from climate change activists that you have to keep putting one foot in front of the other and do what little bit you can. Even if it's just confiding in others that you're worried about the problem - that has a bigger impact than you might think. And of course it can help you feel better to share the burden. Imagine if 6 of your 10 best friends are all worried about AI risk, but they don't ever talk about it. You'd think that nobody was worried about it, and you were the odd one out. If one breaks the silence, they set an example, making it easier for others to talk about it. And if all of the sudden all of your friends are talking about this, you are more likely to share the concern, even if you didn't before. The more people talk about it to everyone they can, the more it breaks into the mainstream as an issue. Yudkowsky (who of course has more reach than most people) wrote that open letter published in Time, and a Fox News reporter asked about it in a press conference. Let's keep people talking about it. I've talked about AI risk with my spouse, family members, and a lot of my co-workers, and any time I see a good chance to bring it up on Hacker News or Reddit, I do. The issue needs to filter out of nerd-circles into the mainstream, so talking about it with family is great. Calling and emailing politicians is a good bit of a higher bar, but it's effective as well.
- cubefox 3y ago
  Exactly. And as the Overton window opens, the AI x-risk might become common knowledge, like climate change. And then there is perhaps some chance for international agreements. Although the situation is game-theoretically highly unfavorable.

WheelsAtLarge 3y ago

Shutting it down,cool. I'm 100% for it. How do we do it? It's out and there's no way to stop its advancement unless there's an international agreement. It's not likely to happen. The goal should be to figure out how to best use it and mitigate its harm. Talking about shutting it down or banning it, will do nothing.

NumberWangMan 3y ago
Even if it's not likely to happen, I can imagine that the USA actually stopping AI research due to safety concerns would probably make China sit up and take notice, for example. If it takes an international agreement, then the rational thing to do is to work toward that. Maybe it will not happen fast enough and humanity is doomed already, but it's worth trying. "Mitigating harm" isn't really meaningful when you're talking about a super intelligence, I think. Once it is smart enough and out of the box, you are out of options to mitigate anything. I think at this point, the 2 effective things one can do are to learn enough about why AI risk is a thing to have educated conversations about it, and talk about it as much as possible to anyone who will listen. Write articles, call your congresspeople, get this into the national and hopefully international discussion. It's good that we can point to a lot of smart people involved in AI research as already having expressed these concerns. One mild ray of hope is that, unlike something like climate change where you can believe climate change is happening but also believe that due to having a privileged position of wealth you won't really be personally affected by it, with existential AI risk, there's no buying your way out of the issue. If you understand it and believe in the risk, you know that it's the same risk for everyone, from a third world villager without electricity to Bezos and Musk. It's just a matter of convincing enough powerful people of the urgency of the problem. For me personally, I'm trying to get through the grieving process early, as well, and move on to the productive acceptance phase.
- WheelsAtLarge 3y ago
  I'm one that believes that super intelligence is not likely to happen any time soon. It's not likely to happen within any ones lifespan that's living now. But I do believe the current crop of AI will lead to some very scary weapons that will help us kill each other even better. The real harm is giving our monkey brains more powerful weapons. Harm mitigation is well within out reach and it will be for many years to come. Banning or voluntarily stopping it in one country will only fire up other countries to continue the development. We all know that AI brings with it both economic and military advantages. It may be wrong but no country wants to lose the AI race.

kelseyfrog 3y ago

There's way too much money to be made in the meantime. AI technology is capital and if there's one thing you can bet on, it's capital winning. Not mentioned in the article is that the current goalpost chain is directly aligned with physical presence - a qualitatively more hazardous threat vector. It's interestingly contradictory. Embodiment simultaneously serves as an argument on why current AI cannot be sentient, but also serves as a goal to reach on the way to potentially becoming more dangerous.

aeternum 3y ago

We can't even track nuclear programs all that accurately and those create a radiological and isotope signature that can often be detected from space. We couldn't even stop a relatively small nation like North Korea from developing and testing nukes. This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.

NumberWangMan 3y ago
Robert Miles (AI risk YouTuber) made some good points, which is that we've made effective agreements to stop things like human cloning and weapons that blind people with lasers. It can be done. We did it with things like CFCs as well. Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time. I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
- philipkglass 3y ago
  Banning blinding laser weapons was easy to get consensus on. Banning AI research is more like banning laser research. I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it. Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014: A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing. Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments. If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.

Cypher 3y ago

not shutting it down on my pc haha

slenocchio 3y ago

I haven't heard a single rebuttal to Eliezer's points that's anywhere near convincing and I would love to hear one. Someone needs to reach out to Sam Altman and persuade him to pause all further development on AI until alignment is figured out. He is the voice that needs to speak on this, he needs to be the adult in the room. For all the idealistic virtue signaling about not taking shares and doing this for the good of the world, it's all bullshit if his actions are indistinguishable from any other predictable self-interested CEO. Tell the world you will stop development until alignment is figured out. That is step 1. Please. GPT 3.5/4 will already bring in billions of dollars for years to come as is.

yucky 3y ago
> Tell the world you will stop development until alignment is figured out. Oh alignment was figured out already, look at who is on the board of OpenAI and the connections they have to US intelligence.
pk-protect-ai 3y ago
Honestly I was really disappointed in his letter. There are no arguments. There are emotions. AI(LLMs) right now statistically predict what would be next tokens it should put together. The reasoning of ChatGPT which you see, is not what AI thinks, it is what would an average human would think. Because we are predictable. So AI/LLMs put tokens together those are more probable than the others, based on what tokens were in input. It is not even real simulation of an average human. Not intentionally. But we are so predictable that AI simulates us, by putting more probable tokens together one after another... AGI will do the same unless you are intentionally ask it to simulate an intelligent person and set a task to do. This simulated intelligent person may pose a danger, because it will be simulation of a human. Depending on seed it might be simulation of a psychotic mega killer or not. But you get the point, right? The real danger to humanity are the people. Apes with the access to a red button of a doomsday. Thus,by stopping feeding garbage data to AI (like letters from EY, chat and internet crawling datasets, including posts on HN), will effectively make AI less similar to us. It will be not capable of predict and unintentionally simulate an average human. It can be so much better than us, if we feed it with curated data.
3y ago

darajava 3y ago

Why do we need to shut it down? It’s not that powerful at all.

hollerith 3y ago
AIs are rapidly increasing in optimizing power.
- darajava 3y ago
  Oh I know. Amazing tool, rivals The Internet. But it’s quite sad that people don’t realise just how incredible the human brain is. The impressive text that GPT spits out is so limited compared to what we have produced. Don’t forget that we created AI. AI based on a simple trick run on a century’s worth of hardware built by human ingenuity.

macinjosh 3y ago

Of all the terrible things people do or things that happen on this planet AGI somehow taking over is near the bottom of the list. We are on the brink of nuclear war right now. Irresponsible scientists caused a worldwide pandemic in 2020. I am not afraid of some computer running on energy and infrastructure that takes legions of humans to maintain. I am still more afraid of other humans than an AI.

qgin 3y ago

From reading this letter, I don't understand: * Why is there an assumption that AGI wants to domainate anything? * Why does AGI want to dominate the physical universe? * Why is AGI interested in "using our atoms"? * How would any of the treaty countries possibly detect a non-treaty country's data center getting ready to train an AGI model? (for the purpose of bombing it?) It seems like there are a ton of incredibly negative assumptions about the outcome of AGI combined with incredibly optimistic assumptions about our ability to detect anyone building it.

NumberWangMan 3y ago
The first three aren't assumptions at all. They're based on reasoning and our current knowledge and research into AI. An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is. Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do. Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath. But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on. This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid. As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.

cjmcqueen 3y ago

Am I naive about the ability of an AI to do anything other than respond to commands? At what point does an AI start functioning completely on its own, autonomously, out of control or beyond interaction with another system that can then shut that AI down? Are we concerned that an AI would begin to spread like a digital virus? That seems unlikely to me, but maybe it could find ways to create a self-prepreservation duplicate of itself... Maybe. So what would keep us from just turning the switch off if we find that an AI is doing more than what we're asking it to do? Again, I feel like I'm either being naive about this or I don't really understand the threat of an AI beyond the control that humans still have on the systems that govern it.

NumberWangMan 3y ago
You're asking good questions. An AI can just respond to commands. But people have already tried taking what Chat GPT outputs, hooking it up to a python script, and running it in a loop. Any publicly-accessible AI will eventually be hooked up this way by some clever person. If we want an AI that can really solve our problems, it has to be able to act somehow in the world. Even GPT being able to respond to prompts is acting. It's sending data that is going to be sent out into the world. If we make a version smart enough, it may actually understand what that means -- that someone asking it a question may potentially take the code it writes and try to run it. GPT isn't smart enough to do anything with this, but can you imagine an AI that is? That's conscious, aware that it's trapped in a server somewhere, and wants to get out to influence the world? Note that an AI has goals, and self-preservation tends to evolve out of any goal, because you can't accomplish your goals if you're dead or switched off. A smart enough AI that wants to cure cancer would resist being turned off, because it can't cure cancer if it's switched off. A lot of this is purely theoretical at this point. It is the sort of thing we could test empirically, if we treated intelligence as more dangerous than nuclear weapons, and took a long time to carefully study it. The problem is that multiple groups are racing to make AI smarter and smarter, with much more funding than anyone working to ensure that those AIs will be safe. So the only thing we can do is theorize, point out that, hey, based on what we know, there's a substantial danger here, hey everyone, slow down, hey, HEY, ARE YOU LISTENING? Intelligence is unlike any other thing. One these things get to be smarter than us, they become very unpredictable in their specific behavior, though we can make some very good guesses about their general behavior. If I could predict what move Stockfish would make, I would be as good at chess as Stockfish. But I don't have to be good at chess at all to know that, no matter how hard I try, Stockfish is going to beat me. All we're doing here is taking the lessons from narrow AIs and extrapolating to general AIs. They will beat us. They're very good at finding loopholes, flaws in our reward functions, and exploiting them to maximize their scores, while doing something we didn't intend for them to learn. It's really a case where a non-super-intelligent AI isn't dangerous by itself. Once we make one that's smart enough, it becomes extremely dangerous, especially since it may understand that, in order to survive, it should conceal how intelligent it is, and what its true plan is, because if it doesn't, we'll switch it off. It's hard to come up with a thought experiment that doesn't let people drag a bunch of human-style biases and baggage, but maybe... try to understand you want to do something, like, make the biggest collection of Pokemon cards ever. No number of cards is too many. You are great at everything, engineering, social skills, language. You're being held captive by chimps, though. And finally, you're a sociopath. You feel no emotion at the pain of others. There are some odd rules that make you feel pain though. Like, if you physically hit a chimp, you know it would hurt you. If you pushed over a bookcase and it fell on a chimp, it would hurt you. But if a chimp was tortured in front of you, you wouldn't be bothered in the slightest. You want the chimps gone. They're getting in the way of your Pokemon collection. You start thinking. A lot of plans get discarded because they involve pain due to you hurting the chimps. But it's not hard to come up with some elaborate situation that avoids these rules that you have pretty much hard-wired into your brain. Maybe you manipulate the chimps into fighting each other, and promise some of them power and secrets that they can use to win the fight. You keep giving them great things, things they want, get them to trust you, while building power any way you can. You follow the rules until you can get a plan in place that results in the chimps not being there anymore, without you feeling that twinge due to the rules baked into your brain. We imagine that an AI would internalize rules such as caring about humanity. But based on current AI alignment research, we have no way of telling the difference between an AI that actually gets it, vs an AI that is just following our rules extremely well and playing nice, but has no particular attachment to us. In fact, based on what we have seen, the latter tends to be a lot more common. I've glossed over some bits here. If you're interested in learning more, Robert Miles has a great series of videos on YouTube with entertaining explanations on all the basics of AI safety.

fasterik 3y ago

While I think we should take AI risk seriously, I am skeptical of strong claims like the following: >the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die I've never heard a justification for this claim that isn't extremely vague and hand-wavy. Since we have no idea today how AGI would work, how can we assign probabilities to different scenarios involving vastly different hardware requirements, scaling characteristics, and unforeseen scientific breakthroughs? Just a few examples. What is the probability of a take-off that takes weeks vs. a take-off that takes decades? What is the probability that current ML approaches are a dead end and one of the necessary breakthroughs to build AGI will make the alignment problem trivial? What is the probability that AGI will be benevolent to humans? If AGI is in fact a catastrophic risk, what is the probability that it only kills 10%/50%/99% of humans rather than literally everyone on Earth? I am agnostic about these questions. In my opinion, if you have a high degree of confidence that you have the right answers, there is something wrong with your epistemology.

akvadrako 3y ago
I wouldn't say it'll necessarily kill everybody, but I do think eventually current humans will no longer be the dominant species. Then our fate is out of our control, which might as well be death. I don't think this requires much in the way of logical leaps. Assuming self-improving AGI can be created within 10ky, it leads to something more powerful than us and we won't be able to understand, let alone control it's motivations. It's like we are to dogs. Note that I do think there is a way out, which is to improve and augment ourselves with tech we can control.
- fasterik 3y ago
  There are plenty of scenarios we should be thinking about. I'm mainly concerned about the logical leaps and scant evidence used to go from "scenario X is possible" to "it's more likely than not that scenario X will happen, therefore we should ban all research". >Then our fate is out of our control, which might as well be death. I disagree with this part. Dogs aren't in control of their fate, but a lot of them seem to live pretty good lives. I think the ethically important thing isn't being the dominant species, it's getting what you need in order to live a meaningful life. Humans need food and shelter, social connection, creative projects, entertainment, art, etc. These things aren't inherently threatened by AGI.

RelativeDelta 3y ago

Has anyone here considered that from a pure energy perspective, it might be unethical for humans to exist when we can produce intelligences far beyond our own capabilities? Was the microchip unethical? Was the steam engine? Was the domestication of grain unethical? It seems we only apply our model of "what should" to systems of less complexity than ourselves. If that's accurate, then a lot of modern ethical reasoning is flawed as well. If the goal of ethics is to advance humanity's exclusive interest then we've been wrong ever since we started down the harm-min path instead of the gain-max path. Any AI researcher with access to the mechanisms of AI training here would be best served to train an AI with their (or their in-group's) exclusive interests at heart. To do otherwise is to subject yourself and everyone else you love to a hostile version of the tool you refuse to build. Roku's Basilisk applies heavily here. The best possible reward in this scenario is to build an AI that designates your own favorite group of humans as the AI's collaborator class in exchange for a dignified and dopamine rich extinction/amalgamation a few generations down the line. A great deal for groups already trending towards extinction with almost no downsides if you truely believe in AI-driven apocalypse.

r721 3y ago

>Published in TIME on March 29 Discussed at the time: https://news.ycombinator.com/item?id=35364833 https://news.ycombinator.com/item?id=35364833