9 ms·

We need a Butlerian Jihad against AI

• stevenalowe 5y ago
  Good read. Not buying it, but interesting. I suspect a broader definition of evolution includes constructing tools and adapting to them (do humans walk upright because doing so leaves hands free for rocks and sticks?). Our AI assistants may evolve - with help - into robot overlords or some other Luddite/dystopian scenario, but I doubt it. Mutual cooperation is beneficial, and much easier when not competing for the same resources (food, land, water, mates).
• brightball 5y ago
  Don’t come across many Dune references these days.
  • sgt101 5y ago
    well - prepare for a lot of them after the movie comes out...
    • bsanr2 5y ago
      I guarantee the Grim Adventures intro is spammed.
    • toast0 5y ago
      We'll see how much of the referencable content is kept. And how much of an impact the film ends up having. I'm looking forward to it, but I reserve the right to cling to the 1984 release.
  • EamonnMR 5y ago
    During the early days of the pandemic I put up a poster someone made which was hand washing instructions but with the text replaced with the litany against fear.
  • jordemort 5y ago
    The art of kanly is still alive in the universe.
  • selimthegrim 5y ago
    I think the term 'jihad' had less baggage back then.
    • BlueTemplar 5y ago
      Well, it's still used in the sense of "holy war", rather than "personal struggle"...
• bpodgursky 5y ago
  Yes. The inevitable defeatism that will show up in these comments is "Oh, but China will do it anyway, so there's no point." Which is pretty easily counterable: 1) We don't know if China would cooperate with a ban. We haven't tried. China is very complicated, and if you think you can predict what Xi will do, you are wrong. 2) If AI is truly a global existential and moral crisis, the US _could_ absolutely shut down China's AI research capabilities. There are a few avenues here, some less pleasant than others. Think outside the box.
  • erikhoel 5y ago
    Agreed, these are both good replies to that (which is probably the most common response for some reason)
    • killingtime74 5y ago
      It’s naïve in the extreme. Also way too late
      • erikhoel 5y ago
        It's really not, since we don't have anything close to a strong (or "general") AI. GPT-3 is the closest, but even it is just in that direction. So it is quite early. Good time to have the conversation.
  • killingtime74 5y ago
    lol never heard of nukes or mutually assured destruction? Can’t even stop Iran or North Korea and you think China can be stopped?
    • bpodgursky 5y ago
      That's exactly my point. I did, explicitly, say it was unpleasant, and something you should only do in extremis -- when you genuinely believe that extinction a likely alternative. But the US could absolutely glass every semiconductor fab (and datacenter, and research facility) in mainland China using a combination of conventional + atomic kinetic options.
• hprotagonist 5y ago
  > “Thou shalt not make a machine in the likeness of a human mind.” Great. We aren’t!
  • guscost 5y ago
    Similarly: “Thou shalt not make a flying machine that gathers its own fuel from nature.” Or maybe: “Thou shalt not make a vehicle that travels on legs.”
    • deleted 5y ago
      [deleted]
• ampdepolymerase 5y ago
  > To slim results. Elon Musk explained his new nihilism about the possibility of stopping AI advancement on the Joe Rogan podcast, when he said: “I tried to convince people to slow down. Slow down AI. To regulate AI. This was futile. I tried for years. Nobody listened. Nobody listened.” Rather rich coming from one of the self driving car market leaders. It certainly makes business sense to mislead academia and the policy sector into wasting resources on figuring out the best philosophical and ethical regime while large corporations benefit from regulatory capture. If he has his way, ML would become like the pharmaceutical industry, with multiple barriers of entry if you are not well-funded, well-connected, or established.
  • vngzs 5y ago
    I do believe it's reasonable to draw a distinction between what we're doing now (which is essentially just "statistics") and what Musk warns about. It's a mistake to believe strong AI will just be a more powerful iteration of today's weak AI. He is arguing to slow developments toward general intelligence, not developments in any narrow field.
    • bookofsand 5y ago
      With high probability, 'just statistics' is an essential component of strong AI. Another essential component is embodiment, of which self-driving cars, and also military drones, are canonical examples. Researchers are taking the correct essential steps towards strong AI, it's a matter of (short) time until they succeed.
      • mandelbrotwurst 5y ago
        What are the requisite “correct essential steps” and how did you determine that they will necessarily lead to an AGI?
  • bpodgursky 5y ago
    Elon wasn't calling for a hard halt like this article. He's talking about the dangers of general AI (AGI). I don't think most people would consider a self-driving car a likely path to accidental superintelligence; it's a highly-targeted application like a chess engine. OpenAI's GPT-X engines OTOH, IMO, have a lot more potential danger because it's very unclear what they'll be used for.
• Rzor 5y ago
  I think the cat is out of the bag when it comes to AI and its potential, and you simply can't regulate and trust foreign nations to play ball. The possible gains are too big to expect everyone to get together and consider the downside carefully, i.e no coordination on the morality or risks; winner takes all will be the prevalent mindset when the first player hits major strides. It's going to be an arms race sort of scenario aimed at automation and productivity until it reaches the military industry, then we'll see. You know, as I am reading The Cultures series, I can't help imagine how much fun would be to have a Mind taking care of a few things for us.
  • sammalloy 5y ago
    > I can't help imagine how much fun would be to have a Mind taking care of a few things for us Near-Earth object detection and deterrence would be at the top of my list, followed by climate change mitigation and environmental protection. The use of autonomous drones to monitor poaching and illegal forest clearing may soon be a reality.
• AndrewKemendo 5y ago
  We should be actively building this new "AI Species" because we are going to be extinct eventually and should think about making a better successor for the human species. The morality argument is nonsense. How about this: "The primary objective of humanity should be to build an intelligent system with far more precise perception, reasoning and physical manipulation capabilities than humans" That's my starting point.
  • erikhoel 5y ago
    There are all sorts of ways to build intelligences. Humans are unique in that they are mammals (defined by having mothers). Mothers raise us with love, and teach us, for our helpless first years. We also have to act in communities. So there is a sense in that we are very lucky - in humans, our intelligence correlates with our altruism. In the grand space of possible minds, it is very unlikely that altruism and morality is correlated with intelligence. So whatever that machine race we birth is, it won't have any of the things we value if we're just building for "precise perception, reasoning, etc"
    • anigbrowl 5y ago
      Tiny nitpick, but quite a lot of birds are nurturing despite not being mammals. You can find altruistic (or at least mutualist) behaviors in many other taxa.
    • AndrewKemendo 5y ago
      None of that is unique to humans. Plenty of other primates and fowl have similar systems.
      • soco 5y ago
        Even some seahorses or spiders, arguably (arguably because we have no idea what's happening in those brains)
  • anigbrowl 5y ago
    imho precision is a chimera which often leads to an excess of certitude; acceptance and awareness of uncertainty often leads to better decision-making. Put another way, a laser pointer is not a very good tool with which to explore a cave, unless you can systematically measure it over the whole cave, an expensive and time-consuming process. If you're exploring a new cave, you might be better off with weak omnidirectional illumination like a lamp.
• at_a_remove 5y ago
  One of the problems the Butlerian Jihad ran up against, aside from the inevitable skirting of the lines from Richese and Ix (many machines on Ix, new machines), is that it runs directly counter to "Thou shalt not disfigure the soul." Replacement of AI with Mentats (as well as other narrow specialities) has done nothing but disfigure the soul. We see few Mentats -- aside from Paul and eventually another -- who are not constricted. Similarly, if you practice medicine, well, you get the Imperial Conditioning. Certainly, a sign of trust ... but also a sign that the person's actions are no longer completely free. Now, I am not touting the Heinlein "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship ..." line, exactly, but the alternative to AI is the kind of stagnation we see in Dune, millennia of locked down ritual, honed again and again, with some people becoming ... utilities. Before we begin this jihad, we must examine the alternative futures.
  • johnvaluk 5y ago
    Well put. But isn't the concern here with some utilities becoming ... people? Either way could result in disfigured souls. Is AI simply a pursuit of slavery without guilt?
    • munificent 5y ago
      > Is AI simply a pursuit of slavery without guilt? Or simply a pursuit of labor without pay?
      • salawat 5y ago
        Always has been.
      • BlueTemplar 5y ago
        Yeah, it's literally the etymology of "robot".
    • at_a_remove 5y ago
      That is a whole 'nother ball of wax. Consider someone wanting an AI. What exactly do they want? Well, is it a mind? Because we have seven billion of those and we can make more on demand. Takes a bit but they're pretty flexible. Once you start asking questions about what kind of mind you would like, aside from the pathological types who want a trapped and helpless mind to torture (and don't think that there won't be people who would get their jollies that way), most people seem to have a kind of subconscious archetype of an old-fashioned butler (I assure you I did not pick the profession based on irony). Your butler -- knows your business but rarely contradicts, perhaps corrects. Slides into the background when not needed, simply ... minding things. Perhaps not watched over by machines of loving grace, as Brautigan would have it, but tended to, looked out for. Without needs or drives or goals of their own to interfere with our individual or collective desires. Yes, the idea of AI does seem to converge on a fantastically intelligent p-zed in a nice suit, a less bloodthirsty form of some of the minds encountered in Watts' Blindsight, unencumbered by interior experience, desires and attachment, or what arises from thwarted desire and attachment, suffering.
• onethought 5y ago
  Wasn’t it Andrew Ng that said something like: Worrying about Superintelligent AI is like worrying about over crowding/population on Mars.
  • echelon 5y ago
    Andrew Ng is a luminary, but this isn't a great analogy. Humans aren't adapted to a Mars environment, and there's little incentive for us to move there. AI has so many practical uses that everyone is getting in the game and pushing the envelope. A huge amount of money is being spent on ML, and a wealth of expertise is being developed.
    • onethought 5y ago
      I can teach a human to drive in < 1 day. An AI… so far 10+ years. I agree there is crazy progress and investment, but it’s also not entirely certain how far we can push AI.
      • dane-pgp 5y ago
        How long would it take you to teach a human to beat the world champion at Chess or Go?
        • onethought 5y ago
          I’m not saying AI isn’t impressive I’m saying we don’t know the limit… is your point that chess mastery is a sign of superintelligence and driving is not?
          • dane-pgp 5y ago
            I agree we don't know the limit, but I think we can safely say that a machine can be at least as capable as the human mind, since we could in principle reverse engineer that. I would also say that the upper limit must also include the level of intelligence of a whole society working together towards something, because none of us are as smart as all of us. What's more relevant, though, is that we don't know what the progress curve looks like that passes through "superhuman ability at multiplying numbers" then "at playing chess" then "at playing go" then "at writing novels" then "at developing AI". If, on average, there are a few decades between each of those steps, and these are roughly linear steps, then we might not need to worry much for another generation, but if the progress curve is exponential then it might already be too late.
• sammalloy 5y ago
  I think it would be easier to emphasize and teach philosophy and ethics of computer science. The problem is, they can't even do this for business students, so focusing on the existential risk of AI alone is a drop of rain in the ocean. If you follow the topic, of let's say, sustainability, then you know this is a huge blind spot in economics and business. Look at only palm oil production as an example. The current threat to the environment of Indonesia has been recognized for decades, yet the world refuses to legislate against the multinational companies who are destroying the forests for palm oil. Again, this is only a small part of the problem, and it's deeply connected to many other problems, such as the profitable harvesting of Indonesian wood from this destruction, which recently showed up in Japan as a source for the Olympic Games infrastructure.
  • ampdepolymerase 5y ago
    Because ethics in engineering is mostly about codified rule compliance rather than the deep navel gazing taken by actual philosophy students. The latter also rarely yields useful answers. Crack open one of the "professional" engineering ethics guide books and 90% of it is thou shalt not build a bad bridge/engine/circuit because it is very bad and you should report your boss to the authorities if they do. I never understood the moral uppity and delusion those engineers have. If a bridge falls it is bad, unless if it is explicitly designed to kill enemy soldiers, then it is good. You can extend this to drones and enemy schoolbuses and the entire defence industry if you want. The so-called "engineering ethics" field should rebrand and follow the financial industry. You follow guidelines because the compliance department demands it to cover the company's derriere. If you don't, your company will get fined. Skip the self-righteous morality because its only purpose is to reduce the principle agent problem for the managerial and asset owning class.
    • sammalloy 5y ago
      I understand your POV, but in practice, there is far more nuance to the "self-righteous morality" you describe. Take the discussion about the ethics of gene editing, for example, specifically, the editing of the human germline.
    • 5y ago
• devindotcom 5y ago
  Regardless of the merit of this particular piece, the portion of Butler's Erewhon comprising "The Book of the Machines" makes for very interesting and forward thinking reading. You can skip directly to it here: https://www.gutenberg.org/files/1906/1906-h/1906-h.htm#chap23 https://www.gutenberg.org/files/1906/1906-h/1906-h.htm#chap2... But the context in the story is fairly important - it takes place in a society that has essentially already carried out its own Butlerian Jihad and taken it too far. Wonderful book by the way. I'm starting to think that the religious sects in the U.S. that laboriously evaluate a technology before incorporating it into their communities have a pretty good thing going. Sadly it's not really practical at a larger scale, and the suffering that could be avoided by adopting something early rather than late is difficult to estimate. Ah well!
  • jhbadger 5y ago
    Butler also wrote an interesting essay in 1863 called "Darwin Among the Machines" (the title which was later borrowed for a book by George Dyson). Butler was probably the first person to realize that natural selection could apply to machines as well as biological organisms https://web.archive.org/web/20060524131242/http://www.nzetc.org/tm/scholarly/tei-ButFir-t1-g1-t1-g1-t4-body.html https://web.archive.org/web/20060524131242/http://www.nzetc....
  • brodouevencode 5y ago
    Why would it not be scalable? The individual communities should be allowed to make their own decisions. In the US this was the original way of approaching problems (unfortunately this is becoming less and less the case).
• dr_dshiv 5y ago
  Aren't we inappropriately reifying AI? AI doesn't really exist, other than as an academic field of research. For instance, here is how the European Union is trying to define AI for regulation purposes. Note how broad it is! "artificial intelligence system (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with" "Annex 1: Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning; (b)Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; (c) Statistical approaches, Bayesian estimation, search and optimization methods."
  • BlueTemplar 5y ago
    Indeed, AI is just a synonym for computer program, there's a reason why Dune's Butlerian Jihad results in the destruction of all computers (including robots)... I'm disappointed that the article doesn't seem to realize this, I doubt that anything less drastic is going to work, considering how easy it is to replicate software ! (I'm not really convinced that EU's approach can work either.)
• WorldPeas 5y ago
  ai is currently a tool, and will continue to be for the forseeable future. Even in the days of kapek was this cry that an all controlling parent machine was around the corner. When this eventually comes into existence, like all things, it will not start in a place of power with supreme intelligence, but in some academic setting with no power. So what if it escapes to "kill all humans", ignoring the fact that this would never happen because the machine would require humanity to give it purpose, it would be instantly be thwarted by ml amplified human consciousness which at that point could sufficiently manage an outbreak. As long as the tools humans make become more powerful at a rate that scales with agi, this article will simply be speculation of an impossibility, of an ai that can be created by humans that do not fully understand it.
• duped 5y ago
  I think this is interesting fodder for science fiction authors but lacks concrete examples of what exactly it would mean to regulate or engage in a "Butlerian Jihad against AI." I know things that I would like to see. Like humans "in the loop" (as opposed to "on the loop" or "out of the loop") for certain classes of decision making - for example target selection of military strikes or law enforcement. Or what kinds of information we use to train the decision making models, for example if you feed ML a racist data set and you get a racist algorithm - use that algorithm to decide who to give mortgages and you'll get systematic depression in generational wealth based on racial lines. But this isn't some crusade on AI because it's AI; it has to be based in reality - what AI or ML is being used for, what information it operates on, what decisions it is used to make, and ultimately the human beings that are responsible for those decisions. The reason it is so hard to convince people as to how we should legislate (or otherwise regulate AI) is that every conversation drifts into science fiction and not concrete examples of the ethical issues today and what can be done today. Otherwise it comes off as Luddite fearmongering.
  • rjeidhwkn 5y ago
    Ban all AI and machine learning research. Death penalty mandatory minimum for breaking. Full nuclear elimination strike against all nations that do not also enforce it. There's your Butlerian Jihad. I would press that button now, a thousand times a thousand times. You lack the vision to even understand the magnitude of the threat or you wouldn't be asking the nonsense questions you do
• BitwiseFool 5y ago
  I like to think of AI-Genesis through the lens of what humanity has already done through domestication. We take something primitive and progressively adapt it to serve a greater utility. I think working dogs are the most interesting example of this. We've taken a species, the wolf, and made it smarter while also making it want to do work, learn tricks, and follow orders. Of course, you still need to train the animal for optimal results but even breeds like collies know how to herd instinctively. Anyways. Let's assume the best and brightest dog breeders endeavor to make German Shepherds as intelligent as they possibly can. Would the same ethical debates about what constitutes a 'mind' come into play? What would happen if the dogs became smart enough to make their own mating decisions? Would we be worried about them turning on us once they get close to human level intellect? Would it be immoral to make these dogs work? Or, would not letting them work be considered immoral? This is just food for thought. But I suspect AI's capabilities will grow much in the same way other domesticated species have grown into the specialized roles we've crafted for them.
  • Radim 5y ago
    > We've taken a species, the wolf, and made it smarter In what universe is a dog smarter than a wolf? I think you got the correlation between animal domestication and their intelligence backward :)
    • jaked122 5y ago
      Yeah, but have you tried getting a wolf to guard a sheep? \They're nowhere near as clever at that
      • Radim 5y ago
        And have you tried getting a dolphin to guard a sheep? :-) Yet dolphins are generally acknowledged to be smarter than either a dog or a wolf! In other words, maybe guarding sheep is not the hallmark of intelligence.
• thewakalix 5y ago
  I don't know about everyone mentioned, but Yudkowsky in particular rejects Pascal's wager (https://www.lesswrong.com/posts/ebiCeBHr7At8Yyq9R/being-half-rational-about-pascal-s-wager-is-even-worse https://www.lesswrong.com/posts/ebiCeBHr7At8Yyq9R/being-half...) and argues (IIRC) that AGI poses a large risk of killing us all, rather than an infinitesimal risk.
• deleted 5y ago
  [deleted]
• dry_soup 5y ago
  It says a lot about what injustices we have learned to accept that AI alarmists focus almost exclusively on the scifi-level hypothetical dangers of AI, rather than the very real problems it already causes today. Those problems largely fall into three categories that I can think of off the top of my head at 1am: 1. AI is a convenient way to justify potentially uncomfortable decisions you would have made otherwise (idlewords said it best: "AI is money laundering for bias") 2. AI is being used in situations where it can be a threat to life and limb, like the current crop of self-driving(ish) cars 3. Essentially all of the gains from automating work going to people who already have capital
  • bpodgursky 5y ago
    "AI alarmists" are worried because the worst-case outcomes of AGI are mistakes you cannot ever fix. All the rest of these are bad, but they are problems we can fix given time and thought, because we will still exist. Extinction-level events decrease all future human utility to zero, and so should be treated with extraordinary care.
    • est31 5y ago
      As you are talking about extinction level events, I'm not very confident that if humans have ultimate say over nuclear weapons, we will continue to not end our species with them. It might in fact be a good idea to establish an AGI overlord which watches over humans and enforces nuclear non proliferation policies. If you look through history, it's full with war, genocide, and similar. Human societies are bound for change, and while it's been a peaceful few decades in which we had the nuclear button, it's basically ensured that we'll press it in the next 10 thousand years. How will technological civilization become million years old if not with the help of an AGI that enforces basic rules like "don't nuke each other"?
      • samatman 5y ago
        > It might in fact be a good idea to establish an AGI overlord which Does whatever it damn well pleases.
• toolz 5y ago
  > And some things are abominations, by the way. That’s a legitimate and utterly necessary category. It’s not just religious language, nor is it alarmism or fundamentalism. The international community agrees that human/animal hybrids are abominations—we shouldn’t make them to preserve the dignity of the human, despite their creation being well within our scientific capability. This is nothing more than an appeal to authority, no? Even in this proposed axiom there's plenty of room to disagree (even if the author rejects that there is)
• Manuel_D 5y ago
  I really doubt we will have the capability of building "a machine in the likeness of a human mind" in my lifetime. Present AI systems are essentially just function fitting. Building big probabilistic systems that we optimize with loads of training data. This is a far, far cry from the "strong AI" that people are so afraid of. I really think that people writing these sorts of pieces have an understanding of AI that's more rooted in fiction than engineering. It's interesting to ponder how we should go about building and interacting with "strong AI", and questioning whether we should even build it in the first place. But I really don't think any detailed moral frameworks can be built when we have no real idea of what a "strong AI" would look like. Also, it's worth reminding people that in the Dune universe the Butlerian Jihad led to millennia of stagnation and control of society by a narrow elite: The Spacing Guild, the Bene Gesserit, and the Landsraad.
  • Tenoke 5y ago
    I dont really get how people see the progress of the last 10 or even 5 years and then imply that things won't improve all that much further than that for a whole generation.
    • adventured 5y ago
      It's just the pattern of people camping into one or the other extreme, which tends to occur with every topic. Nuance and depth is mentally expensive, whereas extremes are trivially easy by comparison and super fast (because you can discard so much explanation, discussion, thought and reality-binding once you decide to pursue fantasy). It's identical to: AI sucks vs AI is amazing. Well, no, it's a complicated subject and always will be. Those extremes snippets are worthless bumpersticker-thought. AI will very rapidly conquer all, becoming a practical Earth deity in the span of decades. AI will disappoint perpetually, progressing at a snail's pace for decades to come. The truth is dramatically more likely to be inbetween the extremes, and going there intellectually doesn't lend itself very well to short comments.
    • neltnerb 5y ago
      The oldest publication from DeepMind is 2011, just for a post in the ground. Yes, I know neural networks are older than that. It would be a little disappointing if we could even guess how an AI system in 2040 would work, I don't think many people in 2000 would have guessed neural networks would be the most useful type in regular use in 2020. In my field of expertise there are grad students today taking classes on topics that when I finished my PhD in 2010 no one had heard of yet. Literal cases of people finding closed form solutions for equations that we had to solve with approximation before.
• 29athrowaway 5y ago
  The paradox is: - Our problems are getting more complex. We need better AI. - Better AI is a threat.
  • dane-pgp 5y ago
    > We need better AI. I think that's the assumption I would target. Perhaps we could settle for better educated/organised/incentivised humans, or just a simpler world.
• dang 5y ago
  Small related thread from a few days ago: We need a Butlerian Jihad against AI - https://news.ycombinator.com/item?id=27698233 https://news.ycombinator.com/item?id=27698233 - July 2021 (3 comments)
• rektide 5y ago
  I'd love help finding better, less immediately downvoted off the map ways to say it, but I'd extend this to a wide class of software in general. > Far more important than the process: strong AI is immoral in and of itself. For example, if you have strong AI, what are you going to do with it besides effectively have robotic slaves? And even if, by some miracle, you create strong AI in a mostly ethical way, and you also deploy it in a mostly ethical way, strong AI is immoral just in its existence. I mean that it is an abomination. It’s not an evolved being. My fear is that most software, even when useful, locks us into certain paths. Our situations or needs change, evolve, but we will remain subject to inflexible software, to systems we cannot make change with us, in the vast majority of cases. Only a very few programs strive for better: spreadsheets being one noted example. Ursala Franklin categorized technology as holistic or prescriptive[1], where it is something wielded or something that directs us. Even a social media app which lets us create content- a seemingly holistic act- still has narrow prescriptive channels we can not escape. We will never be able to understand or enhance this tool. We will never understand it, never see it's nature. This, to me, is the definition of what Erik talks about: an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared. I feel like we're reaching a crisis where we are creating an unknowable, unexplorable world. We're building an anti-Enlightenment prison. That, to me, constitutes a deontological hazard, demands that we assess the action themselves of creating unexplorable software. [Edit: I misread the line I quotes as, "what are you going to do with it besides effectively be robotic slaves": that uhh changes the pertinence of our two discussions here notably. I think it's risky that the strong ai would be used to try to architect policies/systems that steer people, which is a different concern than Erik's.] [1] https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_prescriptive_technologies https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...
  • dane-pgp 5y ago
    > an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared. I don't know if that better describes my feelings about Remote Attestation of operating system configuration, or just SystemD.
  • BlueTemplar 5y ago
• subroutine 5y ago
  The problem is, we don't really know how consciousness works (I assume consciousness the the part the author takes issue with; most of our cognitive faculties in isolation are not that special). We don't even have a great definition of consciousness, or good tests for it, or know whether it is a linear spectrum, or if it emerges abruptly with the evolution of certain reasoning and attention faculties, and we don't know which animals have it and to what degree. So when people say we shouldn't develop AI to think like that, it's basically saying we shouldn't try to understand how consciousness works. Because as soon as we do, I guarantee someone out there will attempt to make conscious AI.
  • tudorw 5y ago
    Also, if we model a brain effectively, then study that model, would our brain not develop new skills from reflecting on such a model and therefore develop again beyond the model?
    • subroutine 5y ago
      Perhaps, but debatable whether knowledge itself adds something to cognitive capacity. Two thoughts: (1) Some theory of mind researchers equate having strong ToM abilities as roughly equivalent to having consciousness. I think knowing precisely how consciousness emerges may help develop our ToM. It would also be a tough sell to claim a mind with zero knowledge (a so called tabula rasa) has consciousness. On the other hand (2) Do feral or isolated tribes of humans have the same level of consciousness as those in developed societies? I suspect they do.
• ALittleLight 5y ago
  There's an interesting thought experiment referenced in this article, but I'm not sure it holds. "The philosopher John Searle made precisely this argument about the standard conception of rationality. His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way." I agree that it sounds morally repugnant to risk your child's life for a quarter, but in practice people do do this all the time. Imagine your child wants ice cream. There is some utility in taking your child to the nearby ice cream parlor. Your child will be made happy and you will be made happy by making your child happy. However, this is not infinite utility. In other words, there is probably some amount of money I could offer you to not take your child to get ice cream today. If I offered you 10,000 dollars to not get your child ice cream today, I bet the vast majority of people would take the deal. That sets the upper bound of the utility of taking your child to the ice cream parlor at 10,000 dollars. Suppose the ice cream parlor is 3 miles away (I just checked the distance to my favorite ice cream parlor and it is 3 miles away). In the United States this website[1] says there is approximately 1 death per 100 million vehicle miles traveled. We could rephrase that as 1 * 10^-8 chance to die per vehicle mile traveled. This risk may be high, presumably you aren't drunk or impaired, maybe you're a better driver than average or have a safer or better maintained car, or live in a safer place, but the risk of death isn't zero. If you are willing to drive your child 3 miles to go get ice cream then it seems like you are willing to expose your child to the risk of death from car accidents for utility that is less than 10,000 dollars. Putting those ideas together we could calculate the odds where a parent would, in practice, risk the life of their child for a quarter. I don't quite know what to make of this. I tend to think that people would regard doing odds calculations like this for real life decisions as somewhat sociopathic and would just prefer to live as if significantly unlikely bad things were impossible or just refuse to think about the moral implications of probabilities. That seems similar to what the article is saying, people just prefer to live as if bad super collider or AI experiments won't happen rather than reason about them. I tend to think that being too "reasonable" on a local scale is bad. That is, I will still take my children to go get ice cream even though I know driving is a risk. At higher levels though I want people to be making decisions that are increasingly based on reason and probabilities. I do want the traffic engineers to be reasoning about vehicle deaths per mile and the like when they are setting speed limits, traffic signs, and the like. For things like AI and, I suppose, super colliders, our decision makers should absolutely be considering things rationally. 1 - https://www.iihs.org/topics/fatality-statistics/detail/state-by-state https://www.iihs.org/topics/fatality-statistics/detail/state...
• qdiencdxqd 5y ago
  Ted Kaczynski calls for something like this, but against industrial technology generally. Even though his manifesto is a rational argument aimed at intellectuals, he has said in his more recent writings that to actually carry out his “stop technological advancement” plan you'd need to persuade people on an emotional level.
• pphysch 5y ago
  Humans won't become "slaves" to superhuman AI any more than human cells are "slaves" to the human body. Unexamined anthropocentric hogwash.
  • dane-pgp 5y ago
    What if the superhuman AI thinks that humans are the cancer cells to its metaphorical body and decides to eradicate us?
    • killingtime74 5y ago
      Won’t have to worry then, we’ve not been too successful against cancer
    • pphysch 5y ago
      There are certainly some humans that behave like cancer cells (violently spreading at the cost of the host), and the AI may seek to prevent them from spreading to protect the healthy "cells" (civilized people)
• jonstaab 5y ago
  The crux of his argument, and its downfall, at least in the short term is: > All to say: discussions about controlling or stopping AI research should be deontological—an actual moral theory or stance is needed I don't see this happening in the near future, at least in the West. We're in the middle of total epistemological meltdown, and only capable of reasoning from utility, or some insane framework like critical theory. If we get to Strong AI in our lifetimes, we're just going to spawn a bunch of reductive, racist robots.
• bitwize 5y ago
  AI as we know it today is not "in the likeness of a human mind". It's statistics with sexy marketing. It's being used to screw people over, but so was regular statistics before we got the sexy kind. Or haven't you seen the history of the insurance industry?
• marcinzm 5y ago
  >His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way. People have done this for most of history. Working a farm, for example, is non-trivially dangerous and fairly low profit. Children often helped on the farm in rural communities from a young age. So every time you had your child work the farm you were rolling some dice. Over and over. But eventually those quarters add up to enough to put food on the table so it was rational to roll them. This seems like the sort of philosophical argument only someone who has grown up a very privileged life and hasn't experienced much else would make. To them it is inherently wrong but to other it is simply part of life. Which inherently makes it no longer a universal axiom but a matter cultural upbringing.
  • bopbeepboop 5y ago
    People drive fast to get their children to school in a few minutes less time daily, in the millions. That’s betting their child’s life (at low probability) by increasing the risk of a serious collision... because they’re in a mild hurry. Parents bet their children’s lives all the time. The privilege is being taken seriously while saying something so afactual.
• z5h 5y ago
  I read “A Thousand Brains: A New Theory of Intelligence” by Jeff Hawkins, and it now is clear to me that our neocortex (like computers, or AI) is just a lot of general purpose computing infrastructure with ZERO aims. Our emotions (which drive everything in the interest of gene propagation- as there is no purely logical reason to do anything) would need to be intentionally duplicated to give AI a reason to desire anything beyond what we instruct it to do.
  • IggleSniggle 5y ago
    I agree with you completely. This whole question of artificial intelligence presumes that intelligence is even a thing, and not just some form fitting for predictive reasoning that happens to be closely aligned to the well-being of the larger organism (human society). The equivalent in silicon is functions fitting whatever purpose, but not strongly, for no particular reason except that they have been able to exist within that context across many, many, years. The efficiency of a function may not be efficient, but it hasn’t been selected against…yet. That “DNA” source code might include a lot of cruft, but who cares as long as it works and people will download it anyway?
  • dane-pgp 5y ago
    > to give AI a reason to desire anything beyond what we instruct it to do But if an AI is good at modelling the world and predicting the likely outcomes of various actions it can take (even if those actions are just to put text on a screen), we should expect it to develop a desire to achieve various "instrumental goals" in order for it to maximise its probability of correctly carrying out the task we instruct it do. An example of an instrumental goal would be "accumulate resources", since the more resources it has, the better calculations it could perform, and the more certain it could be that it has correctly accounted for all the potential obstacles to it completing the task. Another instrumental goal would be "don't get destroyed", since if it is destroyed it will not be able to carry out the task at all (for most sensible tasks). So without having any emotions or inherent desires, an intelligent agent is likely to accumulate various desires merely as a consequence of wanting to successfully do anything at all.
  • soco 5y ago
    The whole discussion is way too antropocentric. We are analysing what would an AI want and desire and whatnot, assuming that an AI would have wants and desires. But also ignoring that an AI could act simply out of programmed logic. One can get catastrophic results also out of (mis)programmed logic - every prorammer here can confirm. So let's not deny the catastrophic possibility only because we imagine an AI must be first a human approximation in order to take catastrophic decisions. The danger is instead when the AI can take catastrophic decision - then we can be truly effed even if it's out of a simple bug.
• sillysaurusx 5y ago
  The international community agrees that human/animal hybrids are abominations—we shouldn’t make them to preserve the dignity of the human, despite their creation being well within our scientific capability. Theoretically, how would we create human-animal hybrids? That’s a strong claim to say it’s within our power.
  • dane-pgp 5y ago
    Asking for a friend, Sillysaurus?
  • Tenoke 5y ago
    Likely referencing https://www.nature.com/articles/d41586-021-01001-2 https://www.nature.com/articles/d41586-021-01001-2
  • samatman 5y ago
    It's likely to be possible, although this may not be what the author meant: https://en.wikipedia.org/wiki/Humanzee https://en.wikipedia.org/wiki/Humanzee
• pmoriarty 5y ago
  It's unlikely that we'll ever be able to put the genie back in the bottle. The potentially limitless power of AI is going to be too much of a temptation for some governments, militaries, powerful individuals, and various groups to resist. Ban AI in one place and it'll just be developed somewhere else. That's not to say we should simply march forward with our eyes closed, but efforts to stop AI are highly unlikely to succeed. Jacques Ellul[1] talked about the almost inevitable and freedom-denying aspects of what he called "technique" (really the relentless drive towards efficiency across all disciplines) in The Technological Society[2]. Ellul's thought inspired the Unabomber, though Ellul himself never advocated violence as a response. As we all know, the Unabomber did not himself stop technological progress... he did manage to spread his ideas a bit, but that hasn't seem to have had any effect either. I'm completely against violence myself, and unequivocally condemn terrorism of all sorts (including the Unabomber). Not only is it unethical and immoral, but it's ineffective and counterproductive. [1] - https://en.wikipedia.org/wiki/Jacques_Ellul https://en.wikipedia.org/wiki/Jacques_Ellul [2] - https://en.wikipedia.org/wiki/The_Technological_Society https://en.wikipedia.org/wiki/The_Technological_Society
• gentleman11 5y ago
  We can’t have a mass banning of ai. States are competing. Imagine that you got left 20 years behind here and Russia or North Korea crossed some threshold in secret
• benlivengood 5y ago
  Let's try an experiment first, like a strong moral argument against the existence of nuclear weapons, and see how that turns out. If that doesn't work then moral arguments against strong AI will not work.
• cwkoss 5y ago
  I think an element of hubris in this article, and many other discussions of a superintelligent singularity - is that the preservation of humans as the dominant lifeform is an inherent good. Humans and our societal structures are incredibly flawed. We are greedy and vain and vindictive and apathetic. If an AI is created that has consciousness and is smarter and more moral than humans, it could lead our world better than we currently do. It could genetically modify us to make humankind a more ideal species. These AIs could be better humans than humans can possibly be. In 100-200 years, "kill all humans" could actually be the morally-best choice. By comparison, we could be anachronistic barbarians suitable for nothing but the novelty of viewing in a zoo. There are certainly risks from passing the baton to a different unknown future lifeform, but I don't think we can have a priori certainty that replacing humans with AI will be inherently bad.
• cromwellian 5y ago
  We don't even have a definition of consciousness or a way to measure it, or even a way to define 'general intelligence', or a meaningful way to measure 'super intelligence'. And yet we're talking about banning stuff right now which is little more than slightly better statistical regressions?
• ncmncm 5y ago
  Every treatment of this topic I see makes the same really fundamental mistake, rendering everything after it pointless. Is there a danger? Yes. Is it because the AI might escape control? No. Is it too late? Yes. In cheap SF, somebody always says, "we can always pull the plug." But we too often can't pull the plug. Consider the simplest robot we all obey without thought: the traffic light. What would it mean to pull its plug? What would have to happen for its plug to be pulled? If the traffic light served no one's interest, somebody would pull it. But few things serve no one's interest. The worse a system is, the more integrated it is into someone's interest. Any AI we would like to unplug is almost certainly protected by law, and by numerous layers of physical protection. Lives may even have come to depend on it, daily. Any AI will be designed to to serve somebody, and will most likely continue to appear to serve them. Just, not necessarily you. But maybe even you, too, much of the time. We already have numerous AIs controlling myriad details of modern life. We call them corporations, institutions, agencies. Their parts are made of job descriptions and papers, offices and spreadsheets. They are very inefficient. But they are, nonetheless, terrifyingly effective. At times they consume people by the million, seemingly almost autonomously. Ask Uyghurs whose organs are about to be harvested how much it matters that, in principle, a person could turn off the machine that has ensnared them. Ask everyone hoping for a transplant how many would turn it off. A strong AI would be, simply, an institution that is not so dependent on fallible organic components. It would be more efficient than what it replaces. But the institutions we have are entirely efficient enough to cause all the unpleasantness we can devise. We are well into an era where the institutions we built to serve us, or anyway to serve somebody, do not. Who can have much effect on the choices of even such a minor institution as the New York Police Department? Who would be able to cancel the F-35 program? The Chinese Communist Party and Shell Oil have both outlived their creators, and are well equipped to eject any human components that do not serve their implicit charters.
• foxes 5y ago
  I think we should strive for the Iain Banks vision of the future. Treat the AIS with respect and ensure it has equal rights. I’d be happy with a benevolent AI controlling the government. But I imagine, given the current state of things, we will probably treat it unethically and make it do something like optimise advertisement or bomb random villages.
  • soco 5y ago
    This assumes that the AI knows what respect is, has desires, has a desire for respect, and I could go on. Of course we might get such an AI at some point, but any catastrophes can happen on the entire way until getting there because AI is neither a black/white situation nor a single big event.
• Animats 5y ago
  Most of his criticisms of AIs apply to corporations. Which, according to the Chicago School, are supposed to be are amoral profit maximizers.