22 ms·

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk https://alexcbecker.net/blog.html#against-ai-risk: The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. It's smart factories that create a vast chasm between a new, tiny Hyperclass and the destitute masses... AI is hardly the only technology powerful enough to turn dangerous people into existential threats. We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. Income inequality is already rising at a breathtaking pace. The internet has given birth to history's most powerful surveillance system and tools of propaganda.

modeless 10y ago
Exactly. The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. And AI is not the only technology with the potential to worsen inequality in the world.
- zevyoura 10y ago
  Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"?
  - singlow 10y ago
    Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.
    - gbhn 10y ago
      True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. In Nuremberg we developed a way to think about this: people outside the central circles of power have a tremendous amount of pressures they are considering. It's definitely the case that some are sadistic and horrible, but more are just following orders and trying to get by as best they can, and punishing them for war crimes is not appropriate. The other side of that coin is that it isn't realistic to expect soldiery en masse to resist illegal orders. It's always more complicated than that.
      - paganel 10y ago
        > Somehow there's always a soldier who will push the button. And then there's always that soldier ready to denounce his camarads for having raped some poor Vietnamese women who had nothing to do with the war itself. Why risk such a PR disaster which might see your funding cut when you can use robots instead? Warzone robots don't snitch on their fellow robots in front of the press and they don't rape, they're only build to kill.
      - MaysonL 10y ago
        > True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. Actually, the unwillingness of the Red Army to do any more invasions was arguably the precipitating factor in the fall of the Soviet empire. In 1988, Gorbachev gave a speech to the Warsaw pact meeting where he told them that the Brezhnev doctrine was no more. No socialist government would be able any longer to count on Russian aid in putting down popular uprisings. A year later, the empire crumbled. Of course, it wasn't the soldiers per se who refused, but the generals who were afraid of the potential mutinies (and the occasional actual ones).
        idlewords 10y ago
        It was Gorbachev who refused, not the generals.
  - zdean 10y ago
    I think they're largely limited by the greater public's threshold for acceptable casualties of their own soldiers during war. What the AI'ing of war does is to remove that natural limit and raise the amount of death and destruction a military can inflict without pushback from the general population.
    - nindalf 10y ago
      I think you're _exactly right_. This is precisely the reason the Iraq and Afghanistan wars proved too costly to prosecute was that it proved too costly in men and materials. The logistics and cost of procuring and moving weapons, vehicles, living supplies etc was high, but manageable. The cost of each body bag which had to be explained to the public was not. Which is why President Obama pivoted to a drone war in areas like Pakistan. Scores of Pakistani civilians are killed on a regular basis for crimes no worse than standing next to a tall bearded man or attending the funeral of a neighbour. The American public has no issues with this because hey, it's cheap to deploy the drones and no American lives are ever in jeopardy. And to be fair about it, apart from the high collateral damage, the Drone War in Pakistan is generally considered to be successful at inhibiting the Pakistani Taliban. This Drone War was the first successful war that we've seen fought without boots on the ground. It's likely that we'll see many more like this as on-board AI improves.
      - wruza 10y ago
        To make big wars like that, you have to have many great robots, not AIs. If ground robots were great at war in a wide sense, we could use it today via remote control / VR interface.
  - PavlovsCat 10y ago
    That's like asking whether we really have to worry about global wars over resources since there are already knife stabbings and car crashes, or whether the Nazis were really that much different from having a bit of a cold. Sadly, being serious about this subject is not one of the strengths of HN it seems, I was kind of spooked when I only got 1 upvote for this: https://news.ycombinator.com/item?id=11685995 https://news.ycombinator.com/item?id=11685995
  - hackuser 10y ago
    > Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"? Currently there's a high cost to it. War is very expensive; even the richest country in the history of the world, the US, doesn't want to bear the costs. And it requires persuading masses of soldiers to go along with it. Roboticized warfare might eliminate those constraints.
  - simonh 10y ago
    No, because their power is limited by their role in society and the organization in which they operate. Even the president of the United States has limited powers. If Obama just decided one day that the best course of action was to nuke Moscow and doubled down on doing so it's extremely unlikely he'd be able to do so. There are enough other people with careers, jobs, pension plans and common decency between him and actually launching missiles that I don't think it's credible that it would happen on a whim like that, even a persistent one. Someone would call a doctor and get the President some medication. However, that depends on the people between the president and nuclear launch being decent human beings that care about law and order and proper procedure. You need to look at the system, not just individuals. Alternatively, let's say someone came to power who genuinely believed nuking Russia was a good option. Rather than order a launch on the spot, what they'd actually do is gradually build a case, appoint pliable or similarly thinking people to key positions, get the launch protocols revised, engineer a geopolitical crisis by provoking Russia and drive events towards a situation in which a nuclear launch seems like a legitimate option.
    - fergie 10y ago
      This is a really strong argument for big government. The smaller our all-powerful military command is, the easier it is for it to go rogue.
      - varjag 10y ago
        It's rather an argument for a crafted system of checks and balances than a big government per se. The totalitarian governments, for example, are absolutely massive, and many don't even have some sort of politburo to reason in a dictator.
      - VexXtreme 10y ago
        A counterargument would be that it's harder for a big government to change direction easily, once decided and set in motion. Wrong decisions can compound because it's easier for a large organization to stay its course, even in the face of increasing harm, as evident by the Vietnam war.
    - 10y ago
  - rocqua 10y ago
    There isn't a whole lot of deniability. Current drones strikes involve explosions and craters.
- Analemma_ 10y ago
  That's exactly what Maciej spends the last third of the article saying: that the quasi-religious fretting about superintelligence is causing people to ignore the real harm currently being caused by even the nascent AI technology that we have right now.
- edblarney 10y ago
  " They will give their human masters practically unlimited power over everyone else." Anyone power that has significant leverage over another power already has that ability. A bunch of 'super smart evil AI robots' will not be able to physical deter/control 500 million Europeans - but - a small Army of them would be enough to control the powers that be, and from there on in it trickles down. Much the same way the Soviets controlled Poland et. al. with only small installations. The 'legitimate threat of violent domination' is all that is needed. So - many countries already have the power to do those things to many, many others via conventional weapons and highly trained soldiers. That risk is already there. Think about it: a decent soldier today is already pretty much a 'better weapon' than AI will be for a very, very long time. And it' not that hard to make decent soldiers. The risk for 'evil AI robots' is that a non-state, inauthentic actor - like a terrorist group, militia etc. gets control of enough of them to do project power. The other risk I think, is that given the lack of bloodshed, states may employ them without fear of political repercussions at home. We see this with drones. If Obama had to do a 'seal team 6' for every drone strike, many, many of those guys would have died, and people coming home on body bags wears on the population. Eventually the war-fever fades and they want out.
- deleted 10y ago
  [deleted]
- bhouston 10y ago
  We don't need ai for massive differences in military effectiveness. That is already here. The us can just destroy most country and substate actors with minimal causalities. The issue is already just difficult matters like differentiating friendlies/neutrals from enemies and not creating more enemies via collateral damage and other forms of reactions.
  - EGreg 10y ago
    The problem arises when non-state actors wreak havoc with drones and AI with inpunity. What would we do if a drone made by a no-name manyfacturer dropped some bombs in Times Square? Who would we blame when someone uses AI to actually sow social mistrust and subvert our existing systems?
    - nine_k 10y ago
      I'm pretty sure that drone-tracking equipment is mounted all over NYC, and more is being deployed as we talk. I can also imagine that a rouge drone arriving from afar and large enough to carry a bomb will be shot down by police if it can't be identified. If it does not carry a bomb, police will apologize.
      - TeMPOraL 10y ago
        I'm not buying that. In fact, the only reason such drone attacks don't happen, or why people hadn't been casually blowing each other up for the past five decades with explosives attached to RC cars / planes is that in general, people are nice to each other. There's plenty of tools out there for dedicated people to wreck havoc in populated areas. Such people simply are very, very rare.
        davedx 10y ago
        Also as recent news from Berlin shows, you don't even need a drone, a regular old van will do
        10y ago
- naasking 10y ago
  > The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. To be fair, it's a small step from effective AI that doesn't malfunction, to an AI over which humans have lost control. It's precisely one vaguely specified command away in fact, and humans are quite excellent at being imprecise.
  - Senji 10y ago
    You can always use LEO EMP nukes to bring us back to the stone age, thus taking out the AI.
    - kabdib 10y ago
      You don't even need to do that; just stop mining coal, or operating oil and gas fields, or scram the reactors. Or more easily, open some circuit breakers; in a pinch you can take down a few electrical towers (not many). A rogue AI's "oxygen" is electrical power, which is really kind of fragile. The emergency power for most datacenters won't last more than a few days without replenishing diesel fuel. For a distributed threat, take out fiber with backhoes. Happens every day now, we just happen to repair it. Stop fixing cuts and it's "Dave, my mind is going...". Of course you have to deal with the AI making contracts with maintenance crews of its own, and do all this before it hardens its power supplies. But our current infrastructure is definitely not hardened against low to moderate effort.
      - ccozan 10y ago
        What stopps the AI to take the path from Matrix?
        krapp 10y ago
        If you mean, what stops an AI from locking humanity into a virtual reality simulation and using its collective body heat as fuel - simple thermodynamics. The Matrix of the original film was originally envisioned as a bootstrap AI, in which the machines were farming humans for their collective processing power - and the simulation that was the Matrix was integral to this purpose, as it served as an operating system for the imprisoned human minds. However, the film's backers felt that concept was too complex for the average moviegoer, and forced the change to "humans = batteries." But, the canonical Matrix would be too inefficient to actually work. The amount of energy needed to maintain a human being over the course of an entire lifetime far surpasses the amount of energy that can be harvested as body heat - and adding some kind VR simulation over that just to keep people who are already physically trapped from "escaping" would just a pointless waste of resources.
- deleted 10y ago
  [deleted]
- erikbye 10y ago
  Don't malfunction? What are you smoking? Everything malfunctions eventually, from software to machinery to cells and neurons.
- otakucode 10y ago
  Human beings have been extremely easy to kill for our entire existence. No system of laws can possibly keep you alive if your neighbors are willing to kill you, and nothing can make them actually unable to kill you. Your neighbor could walk over and put a blade in your jugular, you're dead. They could drive into you at 15MPH with their car, you're dead. They could set your house on fire while you're asleep, you're dead. The only thing which keeps you alive is the unwillingness of your neighbors and those who surround you to kill you. The law might punish them afterward, but extensive research has shown that it provides no disuasion to people who are actually willing to kill someone. A military AI being used to wipe out large numbers of people is exactly as 'inevitable' as the weapons we already have being used to wipe out large numbers of people. The exact same people will be making the decisions and setting the goals. In that scenario, the AI is nothing but a fancy new gun, and I don't see any reason to think it would be used differently in most cases. With drones we have seen the CIA, a civilian intelligence agency, waging war on other nations without any legal basis, but that's primarily a political issue and the fact that it can be done in pure cowardice, without risking the life of those pulling the trigger, which I think is a distinct problem from AI.
  - rocqua 10y ago
    It's about power imbalance and inpersonality. Killing people when neither they nor their friends can retaliate is easier. Being able to say do X or I kill you, without the other party having a defense that will even inconvenience you gives you a shitton of power. Military AI would basically be nukes without the fall-out or collateral damage.
  - mokash 10y ago
    So ideally there would be an AI criminal justice system, just to balance things out.
  - mtgx 10y ago
    This is exactly the same argument of targeted surveillance vs mass surveillance. Saying "humans have spied on each other for centuries" is nothing but a distraction for how far beyond what should be legal mass surveillance is, because it makes it so easy to have everything on everyone all the time. It's nothing like pass targeted surveillance. If anything it's much more like Stasi-style surveillance, but on a much bigger and more indepth scale. And we already know how dangerous such surveillance is with the wrong person leading a country. When war becomes easy and cheap (to the attacker), you'll just end up having more of it. It doesn't help the the military industrial complex constantly lobbies for it either.
- mtgx 10y ago
  This is basically why a lot of people didn't want Google to become a defense contractor while researching military robots. If it did, it would've naturally started to use DeepMind for it. And that's a scary thought.
apsec112 10y ago
There are also nearer-term threats (although I'd likely disagree on many specifics), but I don't see how that erases longer-term threats. One nuclear bomb being able to destroy your city now doesn't mean that ten thousand can't destroy your whole country ten years down the line.
- notahacker 10y ago
  I think the point (which is addressed with Maciej's Almogadro callback near the end) is that the longer term threat being speculated about and dependent on lots of very hypothetical things being true is pretty much irrelevant in the face of bigger problems. I mean, yes, a superpower that had hard military AI could wreak a lot of havoc. On the other hand, if a superpower wants to wipe out my corner of civilisation it can do so perfectly happily with the weapons at its disposal today (though just to be on the less-safe side, the US President Elect says he wants to build a few more). And when it comes to computer systems and ML, there's a colossal corpus of our communications going into some sort of black box that tries to find evidence of terrorism that's probably more dangerous to the average non-terrorist because it isn't superintelligent. Ultimately, AI is neither necessary nor sufficient for the powerful to kill the less powerful. And if it's powerful people trying to build hard military AI, they probably aren't reading LessWrong to understand how to ensure their AI plays nice anyway.
  - FeepingCreature 10y ago
    That's not how risk works. If we want to grow to adulthood as a species and manage to colonize the cosmos, we need to pass _every_ skill challenge. If in 50 years there'll be a risk of unfriendly superintelligence and it'll have needed 40 years of run-up prep work to make safe, then it will do us absolutely zero good to claim that we instead concentrated on risk of military AI and hey, we got this far, right? Considering the amount of human utility on the line, "one foot short of the goal" is little better than "stumbled at the first hurdle".
    - notahacker 10y ago
      I think the article dealt pretty well with risk: you survive by focus finite resources on the X% chance of stopping Y% chances of the near elimination of humanity, not the A% chance of stopping a B% probability of an even worse than the near elimination of humanity event where X is large, Y is a small fraction and the product of A and B is barely distinguishable from zero, despite it getting more column inches most of the rest of the near-neglible probability proposed solutions for exceptionally low probability extinction events put together. I also tend to agree with Maciej that the argument for focusing on the A probability of B is rescued by making the AI threat seem even worse with appeals to human utility like "but what if, instead of simply killing off humanity, they decided to enslave us or keep us alive forever to administer eternal punishment..." either.
OscarCunningham 10y ago
It's possible that we could face both AI risks consecutively! First a tiny hyperclass conquers the world using a limited superintelligence and commits mass genocide, and then a more powerful superintelligence is created and everyone is made into paperclips. Isn't that a cheery thought. :-)
panic 10y ago
It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. This has been done many times by human-run militaries; would AI make it worse somehow? Groups of humans acting collectively can look a lot like an "AI" from the right perspective. Corporations focused on optimizing their profit spend a huge amount of collective intelligence to make this single number go up, often at the expense of the rest of society.
- alexbecker 10y ago
  I don't think AI will cause a paradigm shift here; but like most powerful technologies, I imagine it will have potent military applications.
- citrin_ru 10y ago
  > This has been done many times by human-run militaries; would AI make it worse somehow? Solders in developed countries no longer want to die en masse.
  - LyndsySimon 10y ago
    Soldiers in developed countries no longer have to die en masse. Compare US and Iraqi casualties.
openasocket 10y ago
I'll push back against the idea of smart factories leading to "a vast chasm between a new, tiny Hyperclass and the destitute masses." I mean, if the masses are destitute, they can't afford the stuff being made at those fancy factories, so the owners of those factories won't make money. Income inequality obviously benefits the rich (in that they by definition have more money), but only up to a point. We won't devolve into an aristocracy, at least not because of automation.
- myrmidon 10y ago
  > I mean, if the masses are destitute, they can't afford the stuff being made at those fancy factories, so the owners of those factories won't make money. I think that's beside the point. Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)? It's not like they need money to pay other people (the destitute masses are useless to them). With their only inherent "capital", --the ability to work-- made worthless by automation, the destitute masses have no recourse-- they get slowly extinguished, until only a tiny fraction of humanity is left.
  - openasocket 10y ago
    > Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)? In this scenario, they only need enough factories to meet the needs of themselves and the other aristocrats. Which means there will be massive demand for goods for the common people, which means there will be jobs for people. The only way your scenario works is if the aristocracy produces enough goods for the entire world cheaper than anyone else can, but also making them too expensive for anyone to afford. But that would mean the factories are producing enough goods for everyone despite the fact to very few can actually buy them. Piles of good wasting away outside the factory. The only reason to do this is if the aristocracy was malicious: trying to hurt the rest of humanity despite the fact that there is no real benefit.
    - breuleux 10y ago
state_less 10y ago
The internet also brought us wikipedia, google, machine learning and a place to talk about the internet. Machine learning advances are predicated on the internet, will grow the internet and will become what we already ought to know we are. A globe spanning hyper intelligence working to make more intelligence at break neck pace. Somewhere along this accelerating continuum of intelligence, we need to consciously decide to make thing awesome. So people aim to build competent self driving cars, that way fewer people die of drunk driving or boredom. Let's keep trying. Keep trying to give without thought of getting something in return. Try to make the world you want to live in. Take a stand against things that are harmful to your body (in the large sense and small sense) and your character. Live long and prosper!!!
- tripzilch 10y ago
  which part of the "we need better scifi" slide did you not understand?
ph0rque 10y ago
>We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. And in an almost miraculous result, we've managed not to annihilate each other with them so far. > Income inequality is already rising at a breathtaking pace. In the US, yes, but inequality is lessening globally. > The internet has given birth to history's most powerful surveillance system and tools of propaganda. It has also given birth to a lot of good things, some that are mentioned in a sibling comment.
- blazespin 10y ago
  Yeah but we were never forced into this global boiler room where we're constantly confronted with each other's thoughts and opinions. Thank you social media. It's like there is no intellectual breathing room anymore. Make anyone go mad and want to push the button..
kenko 10y ago
> It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next You know, you don't need to go that far. You know what a great way to kill a particular group of people is? Well, let's take a look at what a group of human military officers decided to do (quoting from a paper of Elizabeth Anscombe's, discussing various logics of action and deliberation): """ Kenny's system allows many natural moves, but does not allow the inference from "Kill everyone!" to "Kill Jones!". It has been blamed for having an inference from "Kill Jones!" to "Kill everyone!" but this is not so absurd as it may seem. It may be decided to kill everyone in a certain place in order to get the particular people that one one wants. The British, for example, wanted to destroy some German soldiers on a Dutch island in the Second World War, and chose to accomplish this by bombing the dykes and drowning everybody. (The Dutch were their allies.) """ There's a footnote: """ Alf Ross shews some innocence when he dismisses Kenny’s idea: ‘From plan B (to prevent overpopulation) we may infer plan A (to kill half the population) but the inference is hardly of any practical interest.’ We hope it may not be. """ It's not an ineffective plan.
- tripzilch 10y ago
  what are you quoting from? who's Kenny? context?
pfisch 10y ago
I mean if you made me a disembodied mind connected to the internet that never needs to sleep and can make copies of itself we would be able to effectively take over the world in like ~20-50 years, possibly much less time then that. I make lots of money right now completely via the internet and I am not even breaking laws. It is just probable that an AI at our present level of intelligence could very quickly amass a fortune and leverage it to control everything that matters without humanity even being aware of the changeover.
yakult 10y ago
There are at least two failure cases here: - a military AI in the hands of bad actors that does bad stuff with it intentionally. - a badly coded runaway AI that destroys earth. These two failure modes are not mutually exclusive. When nukes were first developed, the physicists thought there is a small but plausible chance, around 1%, that detonating a nuke would ignite the air and blow up the whole world. Let's imagine we live in a world where they're right. Let's suppose somebody comes around and says, "let's ignore the smelly and badly dressed and megalomanic physicists and their mumbo jumbo, the real problem is if a terrorist gets their hands on one of these and blows up a city." Well, yes, that would be a problem. But the other thing is also a problem. And it would kill a lot more people.
tvural 10y ago
People are worried about AI risk because ensuring that the strong AI you build to do X will do X without doing something catastrophic to humanity instead is a very hard problem, and people who have not thought much about this problem tend to vastly underestimate how hard it is. Whatever goals the AI has, it will certainly be better at achieving them if it can stay alive. And it will be more likely to stay alive if there are no humans around to interfere. Now you might say, why don't we just hardcode in a goal to the AI like "solve aging, and also don't hurt anyone"? And ensure that the AI's method of achieving its goals won't have terrible unintended consequences? Oh, and the AI's goals can't change? This is called the AI control problem, and nobody's been able to solve it yet. It's hard to come up with good goals for the AI. It's hard to translate those goals into math. It's hard to prevent the AI from misinterpreting or modifying its own goals. It's hard to work on AI safety when you don't know what the first strong AI will look like. It's hard to prove with 99.999% certainty that your safety measures will work when you can't test them. Things will not turn out okay if the first organization to develop strong AI is not extremely concerned about AI risk, because the default is to get AI control wrong, the same way the default is for planets to not support life. My counterpoint to the risks of more limited AI is that limited AI doesn't sound as scary when you rename it statistical software, and probably won't have effects much larger in magnitude than the effects of all other kinds of technology combined. Limited AI already does make militaries more effective, but most of the problem comes from the fact that these militaries exist, not from the AI. It's hard for me to imagine an AI carrying out a military operation without much human intervention that wouldn't pose a control problem. --------- Edited in response to comment--------
- wyager 10y ago
  > if you are given a strong AI randomly selected from the space of all strong AIs Why would this ever apply? We're building them, not picking them out of a hat.
  - AaronFriel 10y ago
    I think the current state of deep neural networking designs and research funding pouring into generalizing the simple neural nets we're working with now suggests that we are, in fact, pulling them out of a hat. Right now we're just discarding all the ones that are defective, at a stupendously high rate as we train neural nets. I can't speak to what method would generate the first strong AI, but I suspect the overall process - if not the details - will be similar. Training, discarding, training, discarding, testing, and so on. And the first truly strong AI will likely just be the first random assemblage of parts that passes those tests.
witty_username 10y ago
Global income inequality has been decreasing. http://voxeu.org/article/parametric-estimations-world-distribution-income http://voxeu.org/article/parametric-estimations-world-distri...
- Ygg2 10y ago
  That's inequality between states. Inequality within states hasn't decreased.
  - witty_username 10y ago
    IIRC inequality of all humans has decreased as poor people have become richer.
mtgx 10y ago
Yes, that's the ultimate threat. But in the meantime, the threat is the military will think the AI is "good enough" to start killing on its own and the AI actually gets it wrong a lot of the time. Kind of like what we're already seeing now in Courts, and kind of how NSA and CIA's own algorithms for assigning a target are still far less than 99% accurate.
tboyd47 10y ago
No doubt that his "inside arguments" have been rebutted extensively by the strong AI optimists and their singularity priests. After all, dreaming up scenarios in which robotic superintelligence dominates humanity is their version of saving the world. That's why I found the "outside arguments" here equally important and compelling. > The outside view doesn't care about content, it sees the form and the context, and it doesn't look good. If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. The best rebuttals to all this are the least engaging. "Dude, are you telling me you want to build Skynet?"
- lmm 10y ago
  > If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. Occasionally the crazies are right. Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk, except turns out they were actually doing this the whole time. The fact that the world hasn't ended tells us virtually nothing about how likely the end of the world is, for the simple reason that if the world had ended we wouldn't be here to talk about it. So we can't take it as evidence, at least, not conclusive evidence. Note also that the same argument works just as well against global warming as it does against AI risk. Turn it around. Suppose there was a genuinely real risk of destroying the world. How would you tell? How would you distinguish between the groups that had spotted the real danger and the run-of-the-mill end-of-the-world cults?
  - tboyd47 10y ago
    The point that I was trying to bring attention to is that one's perception of the AI risk movement changes substantially once you turn your focus from content to form and context. He brings many examples of this (the cult in the robes, Art Bell, etc.) I wouldn't call suspecting the government of surveilling people "crazy." Claiming you know the timeframe for the apocalypse with precision is different. I am from North Carolina -- try a Google Image search for "May 21, 2011." Ray Kurzweil believes he will never die. Balancing content with form and context, what we have here is clearly an atheistic, scientist version of "May 21, 2011."
empath75 10y ago
The real danger of ai is that they allow people to hide ethically dubious decisions that they've made behind algorithms. You plug some data into a system and a decision gets made and everyone just sort of shrugs their shoulders and doesn't question it.
cryoshon 10y ago
what if we made a superintelligent AI that was our Socrates? superintelligence of a military AI is worrisome, but superintelligence of a cantankerous thinking is quite reassuring...
krmboya 10y ago
Isn't that the conclusion he gives at the end of the article? Ethical considerations

modeless 10y ago

Exactly. The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. And AI is not the only technology with the potential to worsen inequality in the world.

zevyoura 10y ago
Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"?
- singlow 10y ago
  Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.
  - gbhn 10y ago
    True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. In Nuremberg we developed a way to think about this: people outside the central circles of power have a tremendous amount of pressures they are considering. It's definitely the case that some are sadistic and horrible, but more are just following orders and trying to get by as best they can, and punishing them for war crimes is not appropriate. The other side of that coin is that it isn't realistic to expect soldiery en masse to resist illegal orders. It's always more complicated than that.
    - paganel 10y ago
      > Somehow there's always a soldier who will push the button. And then there's always that soldier ready to denounce his camarads for having raped some poor Vietnamese women who had nothing to do with the war itself. Why risk such a PR disaster which might see your funding cut when you can use robots instead? Warzone robots don't snitch on their fellow robots in front of the press and they don't rape, they're only build to kill.
    - MaysonL 10y ago
      > True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. Actually, the unwillingness of the Red Army to do any more invasions was arguably the precipitating factor in the fall of the Soviet empire. In 1988, Gorbachev gave a speech to the Warsaw pact meeting where he told them that the Brezhnev doctrine was no more. No socialist government would be able any longer to count on Russian aid in putting down popular uprisings. A year later, the empire crumbled. Of course, it wasn't the soldiers per se who refused, but the generals who were afraid of the potential mutinies (and the occasional actual ones).
      - idlewords 10y ago
        It was Gorbachev who refused, not the generals.
- zdean 10y ago
  I think they're largely limited by the greater public's threshold for acceptable casualties of their own soldiers during war. What the AI'ing of war does is to remove that natural limit and raise the amount of death and destruction a military can inflict without pushback from the general population.
  - nindalf 10y ago
    I think you're _exactly right_. This is precisely the reason the Iraq and Afghanistan wars proved too costly to prosecute was that it proved too costly in men and materials. The logistics and cost of procuring and moving weapons, vehicles, living supplies etc was high, but manageable. The cost of each body bag which had to be explained to the public was not. Which is why President Obama pivoted to a drone war in areas like Pakistan. Scores of Pakistani civilians are killed on a regular basis for crimes no worse than standing next to a tall bearded man or attending the funeral of a neighbour. The American public has no issues with this because hey, it's cheap to deploy the drones and no American lives are ever in jeopardy. And to be fair about it, apart from the high collateral damage, the Drone War in Pakistan is generally considered to be successful at inhibiting the Pakistani Taliban. This Drone War was the first successful war that we've seen fought without boots on the ground. It's likely that we'll see many more like this as on-board AI improves.
    - wruza 10y ago
      To make big wars like that, you have to have many great robots, not AIs. If ground robots were great at war in a wide sense, we could use it today via remote control / VR interface.
- PavlovsCat 10y ago
  That's like asking whether we really have to worry about global wars over resources since there are already knife stabbings and car crashes, or whether the Nazis were really that much different from having a bit of a cold. Sadly, being serious about this subject is not one of the strengths of HN it seems, I was kind of spooked when I only got 1 upvote for this: https://news.ycombinator.com/item?id=11685995 https://news.ycombinator.com/item?id=11685995
- hackuser 10y ago
  > Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"? Currently there's a high cost to it. War is very expensive; even the richest country in the history of the world, the US, doesn't want to bear the costs. And it requires persuading masses of soldiers to go along with it. Roboticized warfare might eliminate those constraints.
- simonh 10y ago
  No, because their power is limited by their role in society and the organization in which they operate. Even the president of the United States has limited powers. If Obama just decided one day that the best course of action was to nuke Moscow and doubled down on doing so it's extremely unlikely he'd be able to do so. There are enough other people with careers, jobs, pension plans and common decency between him and actually launching missiles that I don't think it's credible that it would happen on a whim like that, even a persistent one. Someone would call a doctor and get the President some medication. However, that depends on the people between the president and nuclear launch being decent human beings that care about law and order and proper procedure. You need to look at the system, not just individuals. Alternatively, let's say someone came to power who genuinely believed nuking Russia was a good option. Rather than order a launch on the spot, what they'd actually do is gradually build a case, appoint pliable or similarly thinking people to key positions, get the launch protocols revised, engineer a geopolitical crisis by provoking Russia and drive events towards a situation in which a nuclear launch seems like a legitimate option.
  - fergie 10y ago
    This is a really strong argument for big government. The smaller our all-powerful military command is, the easier it is for it to go rogue.
    - varjag 10y ago
      It's rather an argument for a crafted system of checks and balances than a big government per se. The totalitarian governments, for example, are absolutely massive, and many don't even have some sort of politburo to reason in a dictator.
    - VexXtreme 10y ago
      A counterargument would be that it's harder for a big government to change direction easily, once decided and set in motion. Wrong decisions can compound because it's easier for a large organization to stay its course, even in the face of increasing harm, as evident by the Vietnam war.
  - 10y ago
- rocqua 10y ago
  There isn't a whole lot of deniability. Current drones strikes involve explosions and craters.
Analemma_ 10y ago
That's exactly what Maciej spends the last third of the article saying: that the quasi-religious fretting about superintelligence is causing people to ignore the real harm currently being caused by even the nascent AI technology that we have right now.
edblarney 10y ago
" They will give their human masters practically unlimited power over everyone else." Anyone power that has significant leverage over another power already has that ability. A bunch of 'super smart evil AI robots' will not be able to physical deter/control 500 million Europeans - but - a small Army of them would be enough to control the powers that be, and from there on in it trickles down. Much the same way the Soviets controlled Poland et. al. with only small installations. The 'legitimate threat of violent domination' is all that is needed. So - many countries already have the power to do those things to many, many others via conventional weapons and highly trained soldiers. That risk is already there. Think about it: a decent soldier today is already pretty much a 'better weapon' than AI will be for a very, very long time. And it' not that hard to make decent soldiers. The risk for 'evil AI robots' is that a non-state, inauthentic actor - like a terrorist group, militia etc. gets control of enough of them to do project power. The other risk I think, is that given the lack of bloodshed, states may employ them without fear of political repercussions at home. We see this with drones. If Obama had to do a 'seal team 6' for every drone strike, many, many of those guys would have died, and people coming home on body bags wears on the population. Eventually the war-fever fades and they want out.
deleted 10y ago
[deleted]
bhouston 10y ago
We don't need ai for massive differences in military effectiveness. That is already here. The us can just destroy most country and substate actors with minimal causalities. The issue is already just difficult matters like differentiating friendlies/neutrals from enemies and not creating more enemies via collateral damage and other forms of reactions.
- EGreg 10y ago
  The problem arises when non-state actors wreak havoc with drones and AI with inpunity. What would we do if a drone made by a no-name manyfacturer dropped some bombs in Times Square? Who would we blame when someone uses AI to actually sow social mistrust and subvert our existing systems?
  - nine_k 10y ago
    I'm pretty sure that drone-tracking equipment is mounted all over NYC, and more is being deployed as we talk. I can also imagine that a rouge drone arriving from afar and large enough to carry a bomb will be shot down by police if it can't be identified. If it does not carry a bomb, police will apologize.
    - TeMPOraL 10y ago
      I'm not buying that. In fact, the only reason such drone attacks don't happen, or why people hadn't been casually blowing each other up for the past five decades with explosives attached to RC cars / planes is that in general, people are nice to each other. There's plenty of tools out there for dedicated people to wreck havoc in populated areas. Such people simply are very, very rare.
      - davedx 10y ago
        Also as recent news from Berlin shows, you don't even need a drone, a regular old van will do
        10y ago
naasking 10y ago
> The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. To be fair, it's a small step from effective AI that doesn't malfunction, to an AI over which humans have lost control. It's precisely one vaguely specified command away in fact, and humans are quite excellent at being imprecise.
- Senji 10y ago
  You can always use LEO EMP nukes to bring us back to the stone age, thus taking out the AI.
  - kabdib 10y ago
    You don't even need to do that; just stop mining coal, or operating oil and gas fields, or scram the reactors. Or more easily, open some circuit breakers; in a pinch you can take down a few electrical towers (not many). A rogue AI's "oxygen" is electrical power, which is really kind of fragile. The emergency power for most datacenters won't last more than a few days without replenishing diesel fuel. For a distributed threat, take out fiber with backhoes. Happens every day now, we just happen to repair it. Stop fixing cuts and it's "Dave, my mind is going...". Of course you have to deal with the AI making contracts with maintenance crews of its own, and do all this before it hardens its power supplies. But our current infrastructure is definitely not hardened against low to moderate effort.
    - ccozan 10y ago
      What stopps the AI to take the path from Matrix?
      - krapp 10y ago
        If you mean, what stops an AI from locking humanity into a virtual reality simulation and using its collective body heat as fuel - simple thermodynamics. The Matrix of the original film was originally envisioned as a bootstrap AI, in which the machines were farming humans for their collective processing power - and the simulation that was the Matrix was integral to this purpose, as it served as an operating system for the imprisoned human minds. However, the film's backers felt that concept was too complex for the average moviegoer, and forced the change to "humans = batteries." But, the canonical Matrix would be too inefficient to actually work. The amount of energy needed to maintain a human being over the course of an entire lifetime far surpasses the amount of energy that can be harvested as body heat - and adding some kind VR simulation over that just to keep people who are already physically trapped from "escaping" would just a pointless waste of resources.
deleted 10y ago
[deleted]
erikbye 10y ago
Don't malfunction? What are you smoking? Everything malfunctions eventually, from software to machinery to cells and neurons.
otakucode 10y ago
Human beings have been extremely easy to kill for our entire existence. No system of laws can possibly keep you alive if your neighbors are willing to kill you, and nothing can make them actually unable to kill you. Your neighbor could walk over and put a blade in your jugular, you're dead. They could drive into you at 15MPH with their car, you're dead. They could set your house on fire while you're asleep, you're dead. The only thing which keeps you alive is the unwillingness of your neighbors and those who surround you to kill you. The law might punish them afterward, but extensive research has shown that it provides no disuasion to people who are actually willing to kill someone. A military AI being used to wipe out large numbers of people is exactly as 'inevitable' as the weapons we already have being used to wipe out large numbers of people. The exact same people will be making the decisions and setting the goals. In that scenario, the AI is nothing but a fancy new gun, and I don't see any reason to think it would be used differently in most cases. With drones we have seen the CIA, a civilian intelligence agency, waging war on other nations without any legal basis, but that's primarily a political issue and the fact that it can be done in pure cowardice, without risking the life of those pulling the trigger, which I think is a distinct problem from AI.
- rocqua 10y ago
  It's about power imbalance and inpersonality. Killing people when neither they nor their friends can retaliate is easier. Being able to say do X or I kill you, without the other party having a defense that will even inconvenience you gives you a shitton of power. Military AI would basically be nukes without the fall-out or collateral damage.
- mokash 10y ago
  So ideally there would be an AI criminal justice system, just to balance things out.
- mtgx 10y ago
  This is exactly the same argument of targeted surveillance vs mass surveillance. Saying "humans have spied on each other for centuries" is nothing but a distraction for how far beyond what should be legal mass surveillance is, because it makes it so easy to have everything on everyone all the time. It's nothing like pass targeted surveillance. If anything it's much more like Stasi-style surveillance, but on a much bigger and more indepth scale. And we already know how dangerous such surveillance is with the wrong person leading a country. When war becomes easy and cheap (to the attacker), you'll just end up having more of it. It doesn't help the the military industrial complex constantly lobbies for it either.
mtgx 10y ago
This is basically why a lot of people didn't want Google to become a defense contractor while researching military robots. If it did, it would've naturally started to use DeepMind for it. And that's a scary thought.

apsec112 10y ago

There are also nearer-term threats (although I'd likely disagree on many specifics), but I don't see how that erases longer-term threats. One nuclear bomb being able to destroy your city now doesn't mean that ten thousand can't destroy your whole country ten years down the line.

notahacker 10y ago
I think the point (which is addressed with Maciej's Almogadro callback near the end) is that the longer term threat being speculated about and dependent on lots of very hypothetical things being true is pretty much irrelevant in the face of bigger problems. I mean, yes, a superpower that had hard military AI could wreak a lot of havoc. On the other hand, if a superpower wants to wipe out my corner of civilisation it can do so perfectly happily with the weapons at its disposal today (though just to be on the less-safe side, the US President Elect says he wants to build a few more). And when it comes to computer systems and ML, there's a colossal corpus of our communications going into some sort of black box that tries to find evidence of terrorism that's probably more dangerous to the average non-terrorist because it isn't superintelligent. Ultimately, AI is neither necessary nor sufficient for the powerful to kill the less powerful. And if it's powerful people trying to build hard military AI, they probably aren't reading LessWrong to understand how to ensure their AI plays nice anyway.
- FeepingCreature 10y ago
  That's not how risk works. If we want to grow to adulthood as a species and manage to colonize the cosmos, we need to pass _every_ skill challenge. If in 50 years there'll be a risk of unfriendly superintelligence and it'll have needed 40 years of run-up prep work to make safe, then it will do us absolutely zero good to claim that we instead concentrated on risk of military AI and hey, we got this far, right? Considering the amount of human utility on the line, "one foot short of the goal" is little better than "stumbled at the first hurdle".
  - notahacker 10y ago
    I think the article dealt pretty well with risk: you survive by focus finite resources on the X% chance of stopping Y% chances of the near elimination of humanity, not the A% chance of stopping a B% probability of an even worse than the near elimination of humanity event where X is large, Y is a small fraction and the product of A and B is barely distinguishable from zero, despite it getting more column inches most of the rest of the near-neglible probability proposed solutions for exceptionally low probability extinction events put together. I also tend to agree with Maciej that the argument for focusing on the A probability of B is rescued by making the AI threat seem even worse with appeals to human utility like "but what if, instead of simply killing off humanity, they decided to enslave us or keep us alive forever to administer eternal punishment..." either.

OscarCunningham 10y ago

It's possible that we could face both AI risks consecutively! First a tiny hyperclass conquers the world using a limited superintelligence and commits mass genocide, and then a more powerful superintelligence is created and everyone is made into paperclips. Isn't that a cheery thought. :-)

panic 10y ago

It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. This has been done many times by human-run militaries; would AI make it worse somehow? Groups of humans acting collectively can look a lot like an "AI" from the right perspective. Corporations focused on optimizing their profit spend a huge amount of collective intelligence to make this single number go up, often at the expense of the rest of society.

alexbecker 10y ago
I don't think AI will cause a paradigm shift here; but like most powerful technologies, I imagine it will have potent military applications.
citrin_ru 10y ago
> This has been done many times by human-run militaries; would AI make it worse somehow? Solders in developed countries no longer want to die en masse.
- LyndsySimon 10y ago
  Soldiers in developed countries no longer have to die en masse. Compare US and Iraqi casualties.

openasocket 10y ago

I'll push back against the idea of smart factories leading to "a vast chasm between a new, tiny Hyperclass and the destitute masses." I mean, if the masses are destitute, they can't afford the stuff being made at those fancy factories, so the owners of those factories won't make money. Income inequality obviously benefits the rich (in that they by definition have more money), but only up to a point. We won't devolve into an aristocracy, at least not because of automation.

myrmidon 10y ago
> I mean, if the masses are destitute, they can't afford the stuff being made at those fancy factories, so the owners of those factories won't make money. I think that's beside the point. Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)? It's not like they need money to pay other people (the destitute masses are useless to them). With their only inherent "capital", --the ability to work-- made worthless by automation, the destitute masses have no recourse-- they get slowly extinguished, until only a tiny fraction of humanity is left.
- openasocket 10y ago
  > Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)? In this scenario, they only need enough factories to meet the needs of themselves and the other aristocrats. Which means there will be massive demand for goods for the common people, which means there will be jobs for people. The only way your scenario works is if the aristocracy produces enough goods for the entire world cheaper than anyone else can, but also making them too expensive for anyone to afford. But that would mean the factories are producing enough goods for everyone despite the fact to very few can actually buy them. Piles of good wasting away outside the factory. The only reason to do this is if the aristocracy was malicious: trying to hurt the rest of humanity despite the fact that there is no real benefit.
  - breuleux 10y ago

state_less 10y ago

The internet also brought us wikipedia, google, machine learning and a place to talk about the internet. Machine learning advances are predicated on the internet, will grow the internet and will become what we already ought to know we are. A globe spanning hyper intelligence working to make more intelligence at break neck pace. Somewhere along this accelerating continuum of intelligence, we need to consciously decide to make thing awesome. So people aim to build competent self driving cars, that way fewer people die of drunk driving or boredom. Let's keep trying. Keep trying to give without thought of getting something in return. Try to make the world you want to live in. Take a stand against things that are harmful to your body (in the large sense and small sense) and your character. Live long and prosper!!!

tripzilch 10y ago
which part of the "we need better scifi" slide did you not understand?

ph0rque 10y ago

>We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. And in an almost miraculous result, we've managed not to annihilate each other with them so far. > Income inequality is already rising at a breathtaking pace. In the US, yes, but inequality is lessening globally. > The internet has given birth to history's most powerful surveillance system and tools of propaganda. It has also given birth to a lot of good things, some that are mentioned in a sibling comment.

blazespin 10y ago
Yeah but we were never forced into this global boiler room where we're constantly confronted with each other's thoughts and opinions. Thank you social media. It's like there is no intellectual breathing room anymore. Make anyone go mad and want to push the button..

kenko 10y ago

> It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next You know, you don't need to go that far. You know what a great way to kill a particular group of people is? Well, let's take a look at what a group of human military officers decided to do (quoting from a paper of Elizabeth Anscombe's, discussing various logics of action and deliberation): """ Kenny's system allows many natural moves, but does not allow the inference from "Kill everyone!" to "Kill Jones!". It has been blamed for having an inference from "Kill Jones!" to "Kill everyone!" but this is not so absurd as it may seem. It may be decided to kill everyone in a certain place in order to get the particular people that one one wants. The British, for example, wanted to destroy some German soldiers on a Dutch island in the Second World War, and chose to accomplish this by bombing the dykes and drowning everybody. (The Dutch were their allies.) """ There's a footnote: """ Alf Ross shews some innocence when he dismisses Kenny’s idea: ‘From plan B (to prevent overpopulation) we may infer plan A (to kill half the population) but the inference is hardly of any practical interest.’ We hope it may not be. """ It's not an ineffective plan.

tripzilch 10y ago
what are you quoting from? who's Kenny? context?

pfisch 10y ago

I mean if you made me a disembodied mind connected to the internet that never needs to sleep and can make copies of itself we would be able to effectively take over the world in like ~20-50 years, possibly much less time then that. I make lots of money right now completely via the internet and I am not even breaking laws. It is just probable that an AI at our present level of intelligence could very quickly amass a fortune and leverage it to control everything that matters without humanity even being aware of the changeover.

yakult 10y ago

There are at least two failure cases here: - a military AI in the hands of bad actors that does bad stuff with it intentionally. - a badly coded runaway AI that destroys earth. These two failure modes are not mutually exclusive. When nukes were first developed, the physicists thought there is a small but plausible chance, around 1%, that detonating a nuke would ignite the air and blow up the whole world. Let's imagine we live in a world where they're right. Let's suppose somebody comes around and says, "let's ignore the smelly and badly dressed and megalomanic physicists and their mumbo jumbo, the real problem is if a terrorist gets their hands on one of these and blows up a city." Well, yes, that would be a problem. But the other thing is also a problem. And it would kill a lot more people.

tvural 10y ago

People are worried about AI risk because ensuring that the strong AI you build to do X will do X without doing something catastrophic to humanity instead is a very hard problem, and people who have not thought much about this problem tend to vastly underestimate how hard it is. Whatever goals the AI has, it will certainly be better at achieving them if it can stay alive. And it will be more likely to stay alive if there are no humans around to interfere. Now you might say, why don't we just hardcode in a goal to the AI like "solve aging, and also don't hurt anyone"? And ensure that the AI's method of achieving its goals won't have terrible unintended consequences? Oh, and the AI's goals can't change? This is called the AI control problem, and nobody's been able to solve it yet. It's hard to come up with good goals for the AI. It's hard to translate those goals into math. It's hard to prevent the AI from misinterpreting or modifying its own goals. It's hard to work on AI safety when you don't know what the first strong AI will look like. It's hard to prove with 99.999% certainty that your safety measures will work when you can't test them. Things will not turn out okay if the first organization to develop strong AI is not extremely concerned about AI risk, because the default is to get AI control wrong, the same way the default is for planets to not support life. My counterpoint to the risks of more limited AI is that limited AI doesn't sound as scary when you rename it statistical software, and probably won't have effects much larger in magnitude than the effects of all other kinds of technology combined. Limited AI already does make militaries more effective, but most of the problem comes from the fact that these militaries exist, not from the AI. It's hard for me to imagine an AI carrying out a military operation without much human intervention that wouldn't pose a control problem. --------- Edited in response to comment--------

wyager 10y ago
> if you are given a strong AI randomly selected from the space of all strong AIs Why would this ever apply? We're building them, not picking them out of a hat.
- AaronFriel 10y ago
  I think the current state of deep neural networking designs and research funding pouring into generalizing the simple neural nets we're working with now suggests that we are, in fact, pulling them out of a hat. Right now we're just discarding all the ones that are defective, at a stupendously high rate as we train neural nets. I can't speak to what method would generate the first strong AI, but I suspect the overall process - if not the details - will be similar. Training, discarding, training, discarding, testing, and so on. And the first truly strong AI will likely just be the first random assemblage of parts that passes those tests.

witty_username 10y ago

Global income inequality has been decreasing. http://voxeu.org/article/parametric-estimations-world-distribution-income http://voxeu.org/article/parametric-estimations-world-distri...

Ygg2 10y ago
That's inequality between states. Inequality within states hasn't decreased.
- witty_username 10y ago
  IIRC inequality of all humans has decreased as poor people have become richer.

mtgx 10y ago

Yes, that's the ultimate threat. But in the meantime, the threat is the military will think the AI is "good enough" to start killing on its own and the AI actually gets it wrong a lot of the time. Kind of like what we're already seeing now in Courts, and kind of how NSA and CIA's own algorithms for assigning a target are still far less than 99% accurate.

tboyd47 10y ago

No doubt that his "inside arguments" have been rebutted extensively by the strong AI optimists and their singularity priests. After all, dreaming up scenarios in which robotic superintelligence dominates humanity is their version of saving the world. That's why I found the "outside arguments" here equally important and compelling. > The outside view doesn't care about content, it sees the form and the context, and it doesn't look good. If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. The best rebuttals to all this are the least engaging. "Dude, are you telling me you want to build Skynet?"

lmm 10y ago
> If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong. Occasionally the crazies are right. Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk, except turns out they were actually doing this the whole time. The fact that the world hasn't ended tells us virtually nothing about how likely the end of the world is, for the simple reason that if the world had ended we wouldn't be here to talk about it. So we can't take it as evidence, at least, not conclusive evidence. Note also that the same argument works just as well against global warming as it does against AI risk. Turn it around. Suppose there was a genuinely real risk of destroying the world. How would you tell? How would you distinguish between the groups that had spotted the real danger and the run-of-the-mill end-of-the-world cults?
- tboyd47 10y ago
  The point that I was trying to bring attention to is that one's perception of the AI risk movement changes substantially once you turn your focus from content to form and context. He brings many examples of this (the cult in the robes, Art Bell, etc.) I wouldn't call suspecting the government of surveilling people "crazy." Claiming you know the timeframe for the apocalypse with precision is different. I am from North Carolina -- try a Google Image search for "May 21, 2011." Ray Kurzweil believes he will never die. Balancing content with form and context, what we have here is clearly an atheistic, scientist version of "May 21, 2011."

empath75 10y ago

The real danger of ai is that they allow people to hide ethically dubious decisions that they've made behind algorithms. You plug some data into a system and a decision gets made and everyone just sort of shrugs their shoulders and doesn't question it.

cryoshon 10y ago

what if we made a superintelligent AI that was our Socrates? superintelligence of a military AI is worrisome, but superintelligence of a cantankerous thinking is quite reassuring...

krmboya 10y ago

Isn't that the conclusion he gives at the end of the article? Ethical considerations