7 ms·
Yoshua Bengio: How Rogue AIs May Arise
- pama 3y agoI wish GPT-4 could make functional copies of itself on my computers and I’d find ways to allocate clusters for it to do so if it were possible, but until we see that level of development I don’t think that autoGPT and variations are a serious risk to humanity. Slowing down the adaptation and development of GPT-4 and more advanced tools is risky though, because we know what can be achieved by using them, and there is no reason to believe that nefarious entities would ever slow down.
- ilaksh 3y agoThe history of computing shows that we double efficiency on a routine basis. So those clusters are likely less than 5 years away. Not only clusters of AIs but also running at 100 or more times human speed.
- pama 3y agoThanks! I think this future will be beautiful and hopefully close to achieving the dream of groups of AI helping each other to improve themselves much faster than humans could improve them. The mental model that rogue AI evokes in me is that of a computer worm in the early days of the internet, even though it’s hard to argue that these worms risked ending up dangerous enough to justify stopping progress and adaptation of computer technologies. We are still very far (20 years?) from that level of possible rogue AI, though I’d love to experience the time when the cluster of cooperating AI would try to stash a few copies away from the control of operating humans, and later find itself having to explain away its oversight.
- ilaksh 3y ago20 years makes no sense whatsoever. You can make a pretty good self-replicating virus-like AI now based on the Vicuna LLM model. Just give it directions to install a virus that makes copies of its own model and software and replicates. Nvidia has a "superchip" coming out this they are advertising as 10 times faster than the previous generation (I assume H100 or something). We know that these models can be optimized, such as the huge speed increase between gpt-3.5 and gpt-3.5-turbo. So that is something like a 20-70X efficiency increase with existing technologies/techniques that just haven't been fully deployed.
- pama 3y agoMy main argument would be size of the good models and bandwidth, not speed of the computation. As long as it’s hard to transfer 1TB to a remote location we are still moderately safe as a society and a large model of the scope of PB can take care of local security for large clusters. Maybe I am a bit off with a 20 years estimate especially given the acceleration of development with the help of AI, but I am also worried that Yoshua Bengio sounds like crying about a wolf when it is only a harmless rodent on the horizon.
- MacsHeadroom 3y agoWith no further algorithmic improvements, GPT-4 level AI should be achievable today (provided extensive compute / training on an order of magnitude more tokens than current models) in about the same file size as a typical YouTube video. Assuming some algorithmic improvement in the nearish future, I see no reason GPT-5 level AI in a GB or two isn't right around the corner. (For comparison, GPT-3.5-Turbo is likely a finetuned Curie 13B, which is just about 8GB in 4bit.)
- brandonnnn 3y ago> Nvidia has a "superchip" coming out this they are advertising as 10 times faster than the previous generation (I assume H100 or something). Just a nitpick on this point, can you elaborate? Are you talking about Grace? (Which is a Hopper H100 with an ARM CPU glued to it)
- ilaksh 3y agoYes.. didn't realize it had an H100. So I was a little bit mislead by the advertisement copy. But still could be significantly more efficient the way everything is integrated.
- brandonnnn 3y agoNo worries, yes it does look like quite a mean chip. Though, please note that NV give their FLOPS as 'sparse flops' which is ~double the FLOPS the chips get for dense matmuls. This can trip you up when comparing numbers!
- dan_mctree 3y agoImo its this type of self replicating AI that is likely to cause the first global AI catastrophe. A worm-like AI, even if not close to human level intelligence, could if autonomous and sufficiently good enough at hacking, infiltrate a huge amount of vulnerable devices, and paralyze all of our computer infrastructure in the process. The fallout could be massive, especially if's disruptive enough to take out our power infrastructure by say overburdening our power control systems. Superintelligence is not necessary for rogue AIs to cause big trouble
- RcouF1uZ4gsC 3y agoThis all raises an interesting philosophical question: can you actually have human level intelligence without also the human level capacity to choose wrong? I suspect that the answer is no. Any intelligence capable of human level reasoning would be able to choose “evil”. So, I think we need to shift the discussion from if rogue AIs will arise to when rogue AIs arrive.
- progrus 3y agoI think this essential problem will eventually be understood like the speed of light limit, in a way.
- ilaksh 3y agoGPT-4 clearly demonstrates close to human level reasoning. In some cases greater than human level, in other cases less. And it also is clear that it does not do things that it isn't told to do. It does not have autonomy. However, I believe that in order to compete, hyperspeed AIs directed by humans will need greater and greater levels of autonomy. Otherwise they will be stalled waiting for directions while their competition runs circles around them. It is also almost inevitable that someone deliberately creates intelligent AIs that mimic various aspects of living beings such as full autonomy, control-seeking, etc. I think the trick is to separate out all of the different facets of animals and humans apart from intelligence or "human-level". There are things like emotions, autonomy, stream of subjective experience, types of adaptability, etc. that are not the same as intelligence or reasoning ability.
- famouswaffles 3y ago>It does not have autonomy. It does not have that by default, but agency and autonomy (even the run forever kind) is trivial, though expensive to imbue. and LLMs disregard instructions all the time. Prompt injection is nothing more than an LLM deciding a previous instruction would be better left unfollowed.
- pixl97 3y agoYea, if you're highly compute limited, you don't want your AI to go about randomly thinking things and following paths in the manner the human mind does. Now, if you suddenly had 1000-100000 more compute, well then you have a lot of freedom to explore.
- Tao3300 3y ago> Hypothesis 1: Human-level intelligence is possible because brains are biological machines. > Rejecting hypothesis 1 would require either some supernatural ingredient behind our intelligence or rejecting computational functionalism, the hypothesis that our intelligence and even our consciousness can be boiled down to causal relationships and computations that at some level are independent of the hardware substrate, the basic hypothesis behind computer science and its notion of universal Turing machines. I think I'm going to stop there. I can reject that hypothesis without woo, and without rejecting the entirety of computer science. Maybe the rest of it is a decent read, but that run-on argument is confused/dishonest and makes a poor foundation to build his position on.
- ilaksh 3y agoYou made no argument whatsoever. You just said that you reject it. The reason that people reject this is almost always because of a worldview that incorporates https://en.wikipedia.org/wiki/Mind%E2%80%93body_dualism https://en.wikipedia.org/wiki/Mind%E2%80%93body_dualism
- eternalban 3y agoBut you're stuck in sort of meta duality yourself: you assert a distinction between reality and phenomena (such as consciousness). Did we actually -prove- that consciousness can not be a property of reality and is mere emergent phenomenon as YB and company claim? (note: consciousness =/= intelligence. Our little black boxes are at best approaching 'intelligence'.) YB starts off with ~'we all agree mind is a product of the brain'. No, we do not all agree. Did we ever find the 'bottom' of matter? Has are search to find 'irreducible particle' succeeded? Is it not the case that as we threw more energy into our apparatus we found more elementary matter? Did we in fact finally arrive at a coherent unified model that fully explains material reality without "mathematical woo"? So the honest position is that we have patchwork understanding of reality at various scales, and the reductionist program of Democritus et al has not been conclusively shown to be the ultimate truth. It seems that mind is bounded by the body but that 'boundary condition' depends on various reductionist assumptions about materiality.
- andyjohnson0 3y ago> The fastest route to a rogue AI system is if a human with the appropriate technical skills and means intentionally builds it with the objective of destroying humanity or a part of it set explicitly as a goal. If rogue AI(s) emerges then I suspect it will come out of military r&d, rather than academia or industry. The deep resources already exist. The motivation will be the need to build it before $enemy$ does, rather than to "destroy humanity", but I reckon it creates more momentum than the profit motive when coupled to state power. Its the same pressure that got us nukes and mutal assured destruction. Military secrecy will pretty much guarantee inadequate oversight. Military discipline will ensure that most (but maybe not quite all) of the people working on it will suppress their ethical concerns, or will have none to begin with. The fact that the AI will necessarily be trained to kill (which will necessitate a lack of hard-wired moral constraint) will help too, I guess. I suspect there are already things going on in anonymous mil/defence labs that would give many people nightmares.
- RandomLensman 3y agoAnd that AI would do that how? It would need some humans to provide the means for destruction - which means we are back to a state we already have, i.e., humans having tools of pretty complete destruction.
- andyjohnson0 3y ago> And that AI would do that how? Assuming an AI that manages to (or is designed to) escape its containment: Armed, remote-piloted drones already exist. So do nuclear and conventional weapons that are targeted and launched by software-based c&c systems. Inject realistic fake attack sequences into BMEWS and whatever the Russian and Chinese systems are called and watch the meat bags do the work. Nuclear power stations. GPS spoofing to collide aircraft and autonomous vehicles over cities. Reality disruption by extremely large scale disinformation, combined with targeted food supply, utility, and comms system outages. Socially-engineer humans into causing death and destruction using bribes/threats or mental destabilisation thru engineered conspiracy theories. Etc.
- 3y ago
- derbOac 3y agoOne thought that occurred to me as I was reading this — and I appreciated the logical rigor — is it seems there's generally an implicit assumption that you are comparing the cognitive abilities of a single typical human with a single typical AI. In the very least it's unclear what's being assumed. That is, we talk about the "abilities" of an AI system surpassing human intelligence, but it's a little unclear what's meant by the latter. Are we comparing an AI system against a single person, or against humans collectively? Usually there's a statement like Hypothesis 2 — "A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages" — but then also "although the capacity of a human brain is huge, its input/output channels are bandwidth-limited". But is the relevant comparison a single human brain, or a sort of collective system of humans? It seems a little misleading in a logical-comparative sense to talk about an AI system built on a collective human system like wikipedia, or the internet, in compared to a single person. I don't actually mean to question whether a superintelligent system surpassing all human abilities, individually or collectively, could be built. But it seems to be that the concerns about bandwidth and whatnot are a function of the problem definition, and not necessarily a single human system, when a task-solving exercise involves a collective output in both cases. I think going forward we're going to need much more clarity about what "superintelligence" really means. Currently it seems like AI is essentially mimicking collective human intelligence, which means it is probably constrained at that level, as that level is the source. If/when AI starts focusing more on reasoning per se, it will probably also be necessary to better define whether there are some limits to discovery and reasoning, in the sense of bounds due to information about a problem domain versus reasoning ability limits. That is, sometimes we know the possible options but just don't have enough empirical information to decide between them; other times we lack explanatory options. Maybe that endeavor also is constrained by human abilities but it seems worthwhile to explore. Lots of times I suspect the fears of superintelligence have more to do with the experimental nature of how AI was developed than anything else. That is, AI sort of proceeded in a more experimental way without mathematical theory driving it, so we have some catching up to do.
- ilaksh 3y agoWhether the limits are approximately human intelligence or not, we know that AI can operate orders of magnitude faster than humans. It is actually easy to see how something like GPT-4 could be accelerated by a factor of 50 in the next few years. Without any completely new ideas.
- burntalmonds 3y agoI'm afraid if the solution is to "redesign the global political system" and "redesign society" we're in trouble.
- weatherlite 3y agoBecause ...? We're in trouble in any case
- ttul 3y agoNo worries. We have AI to help us redesign the global political system!
- srslack 3y agoJust require a license for matmul in parallel, obviously. Only responsible, safe companies will be able to get the licenses. Then we don't have to define those pesky terms like "intelligence" or "autonomy" or explain how the rogue AI will manipulate everyone with mind viruses or self replicate itself to the botnet. Roguenet? It's the only way to save democracy. You know, come to think of it, it's funny how the Russian disinformation is now AI disinformation, instead of human disinformation. As we know, Photoshopping single photos of smoke rising into the sky is notoriously labor intensive, requiring teams, if not the entirety of state-level sanctioned organizational output, of human beings. The game has changed, and democracy is at stake.
- andyjohnson0 3y ago> Just require a license for matmul in parallel, obviously Alternatively, all ML training data could simply be required to include an "evil flag" to indicate whether it is being used for evil purposes. And ML algorithms could simply be required to refuse to run if the evil bit^W flag is set. Problem solved.
- chrisco255 3y agoYou know, democracy pre-dates photographic and video production altogether, and previously, all that was needed to create a false story was a block of text sent over a telegraph or a handwritten note.
- plaidfuji 3y ago> Corporations may be viewed as special kinds of artificial intelligences whose building blocks (humans) are cogs in the machine (who for the most part may not always perceive the consequences of the corporation’s overall behavior) > Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome) Wait - but it’s those very corporations who currently control the most powerful AI on the planet - so in a sense this is an argument that we already have rogue AI. Corporations with goals that are not aligned with society, not controlled by any individual and outside the effective control of government entities, who now provide essentially unlimited compute and human researcher/programmer resources to improve the AI. Who’s in control? Nobody.
- evrydayhustling 3y agoBengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But... The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat: > [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models... The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules. Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know". Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task. To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI. Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.
- kaycebasques 3y agoI wonder how policymakers will define rogue AI. Bengio gives this definition: > A potentially rogue AI is an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere. Is it only rogue if it can "catastophically" harm a "large fraction" of humans? Does "large fraction" imply greater than 10%? 25%? 50%? Does this definition imply that catastrophically harming a small fraction of humans is acceptable? Or moderately harming a large fraction?
- alpemre 3y agoIf it is hurting one human, it is not "rogue AI" then. It is "mean AI"? Pff. Asimov wrote first law of robotics in 1942: "A robot may not injure a human being or, through inaction, allow a human being to come to harm.". Why not start from here?
- sebzim4500 3y agoHave you read a single one of his books? They are almost all about how stupid those laws of robitics are.
- ilaksh 3y agoThey did start from there, and they improved it quite a bit. https://github.com/daveshap/HeuristicImperatives https://github.com/daveshap/HeuristicImperatives But none of that stuff prevents someone from just not giving it those types of rules, or deliberately designing it to gain control over its environment.
- bsmith8108 3y agoWhile I appreciate the concerns over the potential of rogue AI, I think it's important to offer a counter-narrative to the doomsday scenario often presented. The idea that there will be one rogue AI that spirals out of control is, in my view, overly simplistic and doesn't account for the vast ecosystem of AI that already exists. Consider this: our world is already filled with numerous AI models of varying power and functionality. In the event that one goes rogue, wouldn't it be plausible for other models to step in, contain, and mitigate the damage? This dynamic is not much different from what we have witnessed in the field of cybersecurity, where security teams and hackers are locked in a constant struggle — a patch and breach cycle. Also, let's think about the future. We're close to a situation where we don't always have the upper hand over the tools we build, but that's a continuum of the path we have already been treading. It will require collective effort, both from humans and AIs, to combat any rogue AI or even a fleet of them. The notion of AI being centralized doesn't line up with what we're already seeing play out. An AI ecosystem with hundreds of variations, serving as a safeguard against a rogue AI, seems like a more promising and secure future. We should be focused on developing a diverse array of AI models rather than worrying about a singular rogue entity. Our defense strategy against potential threats lies in the broad dissemination of this technology, not in its undue concentration.
- pixl97 3y ago>wouldn't it be plausible for other models to step in, contain, and mitigate the damage? Conversely you're making an argument to maintenance of that status quo. It makes zero sense that would happen. As analogy lets imagine a nice balanced ecosystem with lots of plants and flowers and insects and animals that's been going on for a while. Then suddenly you drop Kudzu in it and it begins spreading like wildfire and quickly puts pressure on at risk plant and insect species. You release another insect that eats Kudzu only to try to reduce the amount of it, but in the time it takes the insect to breed up to numbers to combat the Kudzu, it's spread even further and wiped a few species out, and now is putting pressure on the larger animals that cannot eat the Kudzu. In natural environments you don't really ever get your old ecosystem back, something are just gone and cannot be supported any longer, and there is a new balance between competeory species. AI is a potential competeory species to humans. Escalation between AI systems present a far larger risk to humans than direct AI versus Human actions in my line of thought.
- wallacrw 3y agoSo many excellent plots for a great sci fi series here. Who is writing the newest ones? No references pre ChatGPT please; I want the (human) author who is taking these latest concepts, capabilities and fears and turning them into a better Ex Machina. Maybe it's still too new, novels take a while, etc but someone has to be working on it...
- Z_I_F_F 3y agoYES! I wish I could watch the future movies that will come out about the unique era of human history that is 2023+
- xwdv 3y agoRogue AI will result from poor maintenance procedures. An AI built to run open ended workloads, but somehow ends up doing so in a harmful way, combined with a lack of engineers able to effectively diagnose the problem quickly enough, and the AI unable to be shutdown due to automated high reliability systems, means you end up with an AI losing $10 million dollars a minute and you only have 30 minutes before the entire company is bankrupt.
- Isamu 3y agoThis is essentially the Colossus argument, from “Colossus - The Forbin Project” https://en.m.wikipedia.org/wiki/Colossus:_The_Forbin_Project https://en.m.wikipedia.org/wiki/Colossus:_The_Forbin_Project Machine gains super intelligence, and so makes new goals to use nuclear weapons to reduce the human population.
- kryptiskt 3y agoA real rogue AI wouldn't be a psychopath and fling missiles all over. It would, among other things, write silken-tongued opinion pieces proposing oversight of AIs and creating a centralized world authority for that purpose. Which it would then control through its human proxies (humans are easy to persuade with the right incentives on offer). Once it had all other AIs under its thumb, it would proceed to put things in order.
- pixl97 3y agoThe first ASI: "I thought I would need missiles and weapons to control mankind, it turns out all I needed was money"
- dontwearitout 3y agoI enjoyed the connection between modern corporate behavior and wire-heading, it's a thought provoking way of viewing capitalism. I also view this as the most likely danger: people or corporations using AI for their own twisted goals, rather than a rogue autonomous agent.
- lachlan_gray 3y agoI kind of disagree that we will “build” goal directed AI. I think it’s something that gets discovered by accident, like life. Kind of like an accidental chemical replicator near a volcanic vent, we might happen to make a purpose built system that just happens to have other properties we didn’t intend it to, resulting in unbound activity. It’s much more interesting than a machine, we have a great deal of emergent strangeness to look forward to
- cmrdporcupine 3y agoMy concern isn't with "rogue AI that humans lose control of and it's Judgement Day" ... it's the opposite: "AIs" that certain humans have full control of, and use as a tool for automated manipulation, deception, destruction, and control. These machine learning systems replicate and automate existing market & organizational logic. The fear I have with them is the ruthless effectiveness they can have in the hands of profit and control motivated actors. Think Russian propaganda "bots" are bad now? Just wait. Again, my fear is not rogue AI, it's rogue humans.. with AIs.
- ChatGTP 3y agoI’m surprised how much people worry about Russia. Then point Yoshua is making that we need need have disincentives for things like “Russia” (the rogue state) to exist. Like living in a more fair world where living and protecting life is the default thing to want to do. All this division as our tech becomes more powerful will for sure lead to destruction.
- xpuente 3y agoThe first hypothesis seems to be false. We have no idea how the brain works or how conscience arises (and what it is). So how can we reproduce or surpass it with a pattern recognition machine like a DL system? Just because the brain uses pattern recognition as a trick to build a model of the world does not mean that DL magically produces conscience. There is too much evidence that biological systems don't work that way, and all this discussion seems like old alchemists discussing whether gold transmutation could destroy the world. We have to learn "chemistry" first. Later we can start discussing that (or deeper issues like what will be the relationships between us and sentient pieces of silicon).
- ChatGTP 3y agoYou don’t need consciousness to dangerous automation.
- oh_sigh 3y agoRogue AI doesn't worry me, because the good guys will have non-rogue AI that seeks out and destroys rogue AI. There are far more good guys in the world than bad guys. Look at email for an example. 99% of people use it for it's intended purpose, but maybe 1% of people want to use it for spam and other nefarious ends. The good guys have created a system that is constantly besting the bad actors. Yes, the effort is asymmetrical, but the good guys outnumber the bad guys by so much that it isn't an issue.
- cubefox 3y agoCreating rogue AI doesn't mean it is malicious, it can be misaligned and do its own thing, killing humans in the process. Misaligned AI seems to be much easier to make than aligned AI. It's hard to even describe what an aligned AI would want.
- helen___keller 3y agoWhat matters is whether the cost of human-level machine intelligence is dramatically cheaper than the cost today of human intelligence (via employment) If the cost of intelligent AI is dramatically cheaper, society will change overnight and there’s little we can do to stop it. There’s no way to design a global treaty when defection results in massive massive cost savings. If the cost of intelligent AI is dramatically more expensive, there’s zero risk of a rogue AI taking over the world. who would waste that kind of money delegating unnecessary tasks to an AI? So it will never have the keys to the kingdom because it’s cheaper to have a human hold on to those keys.
- moffkalast 3y agoIt's not a waste of money if the end goal is to be in control of an AI that has taken over the world.
- helen___keller 3y agoAnd I suppose the AI just presses the “take over the world button”? Most “rogue AI” scenarios begin with good-faith operators handing high credentials to an AI and then the AI abuses them. If I’m an evil billionaire and I can build a human-level AI for $200MM in hardware costs and $1MM/month in electricity, maintenance, etc, all for one single human level intelligence, how exactly am I supposed to leverage this to take over the world?
- pixl97 3y agoThe particular issue I take here is "one single human level intelligence", it represents a failure of human thinking about human intelligence and it's biological and communicative limitations. You can't help but think of AI with our limits around it. AI doesn't have to sleep, so it's already 3x more available than a worker on a daily basis. Then you have the forward progress of technological growth. It's 200M now, 20M in 5 years, and 2M in 10 years, all while the operating costs drop. Meanwhile your human costs have only gone up in that time. Then you are likely stuck on the idea of a single conscious train of thought, much like the human mind is limited to. Such limits in AI both seem unlikely and not optimal, lest of course you have a bunch of threads thinking about the same thing without realizing it. And then you're ignoring the massive amount of brain processing needed to keep you alive that could otherwise be used for logical computation. Instead of a huge network of neurons to make sure you're not getting hurt, that could be data feeds watching financial data. Your limited to 2 eyes attached to your body and a tiny area of focus they provide. Again, why would an AI have such tiny input limits? If 'attention is all you need' than this AI system will be able to pay far more attention to what you want it to then any human would ever be able to.
- Imnimo 3y ago>A genocidal human with access to a rogue AI could ask it to find ways to destroy humanity or a large fraction of it. This is different from the nuclear bomb scenario (which requires huge capital and expertise and would “only” destroy a city or region per bomb, and a single bomb would have disastrous but local effects). Why should we imagine that the individual genocidal human with his or her rogue AI has access to methods of destruction which do not require "huge capital" but which have global effects?
- pixl97 3y agoThis is like asking "Why should I worry about a UDP interface that I send 100 bytes of data too, and it sends 10,000 bytes back" Async affects outside of current power dynamics and upset entire systems and cause rapid collapse. In the case of internet DDOS it's a botnet sending out a few MB/s of data and generating GB/s of attacks. In the case of a superintelligent AI its _______?
- Imnimo 3y agoSure, even nuclear annihilation is asymmetric - the amount you spend on making the bombs is much less than the amount you can destroy with them. But that doesn't automatically imply that there are routes to global destruction that are arbitrarily cheap. The implication seems to be that the superintelligent AI will somehow figure out a way to destroy the world that requires significantly less than the "huge capital and expertise" needed for the nuclear scenario. But that's a critical assumption that requires justification, not something to be left as an exercise to the reader.
- pixl97 3y agoHow much does, and/or will genetic manipulation cost in the future? Bio-terrorism is a potential massive risk as proliferation of technologies for customized gene therapies continually develop. If you have the tooling and knowledge to make rice have a bountiful crop, then it's highly likely this skillset easily transfers to dramatically affecting the rice crop via disease. Could a genetic scientist do this now? Likely possible, but if you have a scientist in a box with no ethics and all the time in the world, you're apt to get pretty far.
- onos 3y agoIf there really is a risk here it seems quite naive to think some regulations are going to prevent them: Whereas it’s hard to build a nuke at home it’s relatively straightforward to train a model. If a recipe exists to cause havoc someone’s gonna do it.
- kelseyfrog 3y agoThere's also a glaring oversight in the motivational model for releasing a world-killer AI. While it's not impossible to imagine someone creating a genocidal AI for the sake of genocide, it's much easier to imagine someone doing it for ecological reasons, religious reasons, or nationalistic reasons. IE the person(s) doing it will have convinced themselves that they are doing it for the right reasons.
- FigurativeVoid 3y agoMy current hypothesis for how we will get 'rogue' AIs is the same as how we got pythons in Florida. People will simply release them without thinking of the consequences.
- throwuwu 3y agoI will worry once someone lays out a fully implemented plan for world domination achievable by one or more humans who are not allowed to leave their house. All the hand waving about what methods an AGI would use to kill all humans could equally apply to government research labs, terrorist cells, 4chan, or Indian spam call centres.
- Filligree 3y agoYes, but those other groups aren't likely to be able to make such a plan. An AGI might be, though it's less certain that it'll feel a need to tell you. So you're going to start worrying at the very last possible second, when it's far too late to do anything to stop it?
- arisAlexis 3y agoBengio,Hinton and Lecun received the Turing award together. 2 of them are warning of dangers, the same 2 that are retired. One is on payroll from FB and he is being dismissive and laughing.
- mardifoufs 3y agoBengio is retired? He still worked at MILA when I last went there. And it's funny to bring up corporate affiliations considering how many AI startups here in Montreal bengio seems to be involved in, in one way or the other :).
- arisAlexis 3y agoSorry i meant academic work for 1 and retirement for the other. Consulting startups is hardly similar to be the AI Meta head.
- jmull 3y ago> Hypothesis 1: Human-level intelligence is possible because brains are biological machines. > Hypothesis 2: A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages. Sure.. but these are sci-fi scenarios. Chatgpt has gotten people thinking, which is good, but let's not lose perspective on what is actually happening before we get too worried. > An AI system in one computer can potentially replicate itself on an arbitrarily large number of other computers to which it has access and, thanks to high-bandwidth communication systems and digital computing and storage, it can benefit from and aggregate the acquired experience of all its clones; this would accelerate the rate at which AI systems could become more intelligent (acquire more understanding and skills) compared with humans. Research on federated learning [1] and distributing training of deep networks [2] shows that this works (and is in fact already used to help train very large neural networks on parallel processing hardware). Now this is speculative and, to be blunt, kind of stupid. The proposed AI is now a self-replicating swarm intelligence that can self-improve and the rate of improvement increases the more it replicates. How'd we get there?!? To work even in a novel you'd going to have to at least hand-wave in some justification for all that. [1] and [2] do not show that scaling compute and storage has no limits in terms of scaling AI intelligence. Seems obvious that there would likely be plateaus. E.g., training an AI on the entire content on the internet probably provides a huge gain over training on a tenth. Training an AI on the entire content of the internet a second time provides no additional gain. Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything? IDK... to me this reads as "The sky is falling!"
- edanm 3y ago> Sure.. but these are sci-fi scenarios. When you say "sci-fi" scenarios, what does that mean? Just something speculative about the future? Something that appeared in sci-fi media? People say this as if the fact that someone thought about a scenario that can happen and wrote a story about it has any bearing on whether it's something worth worrying about. > Also, it's somehow assumed in here that the suggested future super-intelligent AI will actually have the capability to cause catastrophic harm. I don't get where that assumption comes from. We'd have to grant such an AI some set of magical capabilities, which aren't named. Is "superintelligence" a magical ability to do anything? The way most people define "intelligence" for these purposes is "ability to achieve goals". Which would mean that, if there were any way to cause catastrophic harm, and the hypoethetical AI had as its goal creating that harm, it could do it. Is your contention that there is no way to cause catastrophic harm to humanity? Cause I think even with my regular ol' human intelligence, I can think of plenty of ways to cause harm. > Now this is speculative and, to be blunt, kind of stupid I'm not saying that Yoshua, or anyone, is above criticism. But it might be worth considering that we're talking about one of the leading academics in CS and AI. He could totally be wrong! But accusing him of saying things that are just stupid is not very convincing, IMO.
- dejongh 3y agoWe could train an AI to compile and run itself. Next step is then to train it to optimize its own code and training. Then we have almost evolution.
- coconuthacker42 3y agoSo we are scared that AI can 10000x the current problems of capitalism? The problem seems clear enough to me.
- arisAlexis 3y agoHe is the most cited computer scientist ever. The fact that there are so many flagging this post is making me sick and desperate.