32 ms·

Hostile AI: You're soaking in it

130 points by mortenjorck 13y ago

harrystone 13y ago
A blog about the evil corporations, hosted by Google. I love the internet.
zeno334 13y ago
So McDonalds is ok because it is at least somewhat restrained by the need to offer a useful good (food) but the financial industry is parasitical and evil and offers nothing of use? Seriously?
- evv 13y ago
  I think you misunderstand. The author is saying that all corporations are "parasitical" and even "evil", for humans that is. McDonalds would do anything to squeeze more money out of its customers. Like most corporations, they profit from manipulating the human masses.
  - zeno334 13y ago
    Uhm, whatever his other points may be, I can't take him seriously when he lumps all financial institutions together into one group that offers nothing of use. Sure, you can argue all day against the usefulness of speculative speed trading firms - and even then you'd still have some decent arguments against you - but to argue that financial institution in general offer nothing of use is ludicrous. Some relevant quotes: Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end. Of all human institutions, these seem the most parasitical and dangerous. it [the financial system] has interests that are distant or hostile to human goals
    - at-fates-hands 13y ago
      This is the textbook definition of utilitarianism: "Utilitarianism is a theory in normative ethics holding that the proper course of action is the one that maximizes utility, usually defined as maximizing happiness and reducing suffering." He sees financial institutions as a waste, and therefore consuming vast amounts of resources to no useful or positive effect.
    - fennecfoxen 13y ago
      Yeah... it's easy to criticize the financial network. So would you mind if we tacked on an extra N% to your mortgage payment, like you'd expect with less efficient financing? :)
- norswap 13y ago
  How would you say this assessment is wrong?
  - aetherson 13y ago
    Well, it's wrong in no small part because the financial industry does, of course, produce something of value. I don't really get how the hard core "finance is fucked up" people imagine that financial firms got started. Do they imagine they were just created by evil aliens or something? Finance firms provide lots of very sophisticated ways for money to get from people who have lots of money that they don't currently have a use for, into the hands of people who have a need for money they don't currently have. Note this is a weak claim. I'm not saying anything about the efficiency of this process or whether the financial industry is good or evil or anything. But you can't criticize the finance industry without at least understanding and engaging its outputs.
    - Florin_Andrei 13y ago
      Computer, per se, are not bad. Computers infected with malware are bad. Infected computers can still do useful things for you - and in the background they'll be screwing you over also. The argument I would like to make is not that finance is bad intrinsically - as you've said above. The real problem is that the current instantiation of this concept has grown a series of goals of its own, and to fulfill those goals it is doing things that use the common, shared, limited pool of resources of the global economy, siphoning off stuff that the rest of us could put to better use. In other words, they could perform useful services for us, and they do. It's all the extra layer on top of that that's the problem. In a strict, mathematical sense, the OP probably exaggerated when saying they are entirely parasitical.
  - zeno334 13y ago
    Uhm, well, the financial industry isn't parasitical and evil nor does it have interests that are distant or hostile to human goals. It's just another profit oriented industry that offers services which plenty of entities find valuable enough to pay for. The financial industry is as evil as any profit oriented industry can be and to single it out as something completely wasteful and parasitic is unreasonable.
- effdee 13y ago
  "The only useful banking innovation was the invention of the ATM." -- Paul Volcker
- sp332 13y ago
  McDonald's isn't "ok", but it's limited in how evil it can be because it has to give some benefit to other people. A hostile AI could be even worse, because it doesn't have that constraint.
  - zeno334 13y ago
    But surely you can't compare a hostile AI which can do unlimited harm due to this lack of constraint to the financial industry can you?
    - sp332 13y ago
      Why not? A 2012 report estimated over 19 trillion dollars in lost household wealth, and contribution toward the largest number of people living in poverty ever recorded by the US Census bureau. http://www.pbs.org/wgbh/pages/frontline/business-economy-financial-crisis/money-power-wall-street/how-much-did-the-financial-crisis-cost/ http://www.pbs.org/wgbh/pages/frontline/business-economy-fin...
  - randomdata 13y ago
    The financial system is what provides farmers with the necessary capital to be able to grow the food McDonalds needs to be in business. That doesn't necessarily make them okay either, but isn't allowing people to eat, without having to worry about producing their own food, at least of some benefit as well?
    - sp332 13y ago
      Financial institutions aren't inherently bad, but the ones we have are bad.
      - randomdata 13y ago
        Doesn't 'financial institution' cover a pretty wide gamut? There are small institutions that exist in only a small town, to government owned entities, to giant businesses that span the entire world. Are they all categorically bad? What can be done to make them less bad?
deleted 13y ago
[deleted]
NoodleIncident 13y ago
The jab at the financial industry is unnecessary, but the actual point is very interesting. I first heard of this as the 'Santa Claus' argument on a forum thread. Short version: just because you can't point at a single person and say 'he is Santa', doesn't mean that Santa doesn't exist. Despite living solely as a meme in people's heads, he has desires (make kids happy on Christmas, make kids behave), and can act on them (put presents under the tree). Those who supported this position used Microsoft as an example. 'Microsoft', like every company, is at its core an idea or a meme that everyone just accepts exists. Parts of it get written down, of course, and various people hold different parts of it in their head. Nevertheless, the company is its own entity, with its own goals and means of achieving them, yet exists only in people's heads.
- pdonis 13y ago
  But since Santa only exists as a meme in people's heads, Santa can't do anything without the people in whose heads the meme exists doing something. Similarly, corporations can't do anything without the people that own them and work for them doing something. So viewing corporations as quasi-autonomous agents is misleading; they can be viewed as collective agents in some respects, but in the end everything they do has to come down to some human or humans taking some actions. Basically, the author of this article is punting: he's saying the problem is corporations being "effectively independent of human control", instead of actually looking at the actual humans whose decisions and actions constitute what the corporations do.
  - sutterbomb 13y ago
    I don't think he's punting. He's saying that actually looking at the actual humans is misguided, because they can and will be replaced or exchanged with no effect on the system as a whole. At the micro level, a department or even a company may change course in a noticeable way with one person being replaced. But industry wide, and society-wide, the "actual people" are mostly irrelevant to the mechanics of the system.
    - pdonis 13y ago
yukichan 13y ago
If corporations are so evil why has their pay fed my family, paid for my house and created so many amazing products we surround ourselves with. It's ridiculous. Big organizations have problems, but to pronounce them as inherently evil ignores all the good they do. Maybe you have a different idea of good in mind, in which case we'll probably never agree on anything. When people organize they can create amazing things, and that's what corporations are, an organization of people: employees, customers, and investors. And they do amazing things. Go to a country that doesn't do so well at creating functioning organizations and see the difference.
- r00fus 13y ago
  Let's not dilute the point. The article states that the financial industry* is the "hostile AI" not all corporations (in fact, it clearly states that even the hated "energy" corporations actually provide a product - even if that ). Having recently re-read Dan Simmons' Hyperion series (sci-fi humans vs. AI masterpiece), I could almost see the analogy perfectly - finance is key to almost every part of our individual life, business and even political machinations. And in each, the big finance outfits are impervious, indifferent and have their own goals outside of their stated purpose. * I'd expand slightly to include all of FIRE - finance, real-estate and insurance
- dropit_sphere 13y ago
  They're not evil, they're not good. They just are. Blind worship and blind fear are errors of the same category: mistaking their nature.
- ThomPete 13y ago
  I am not disagreeing fundamentally but you can't use your success as an indicator of whether something is good or bad. The corporation that feed you might do other things other places to other people. As an example the De Beers in the diamond industry.
- harrystone 13y ago
  It's just much easier to think of things as being black and white. It really doesn't matter if it's true, it's just comforting.
fennecfoxen 13y ago
Naah, corporations aren't independent profit-seeking AIs. They don't bother making money except insofar as their owners drive them to. Even in that state, they're subject to principal/agent conflicts in which people seek to turn the company's revenue into their own overpriced salaries / stock options. Now, government bureaucracies may be another story, especially once they grow lobbying arms ;)
madaxe_again 13y ago
The point here is that the generation of wealth, which is the conversion of labour and marginal efficiency into a number in a ledger, is not in and of itself aligned with human interests - it turns valuable labour and intelligence into a store of labour-time exchange, which can be used to purchase more labour, rather than any absolute benefit. The problem arises when the store of capital grossly exceeds available useful labour, which it does now in the hands of those who hold it. This causes inflationary forces, the results of which are very visible, and in this way the excess labour-time is simply destroyed, as it becomes devalued. Money is a highly inefficient way of improving the world.
- AndrewKemendo 13y ago
  >Money is a highly inefficient way of improving the world. True vis a vis itself - as money in and of itself does not have an ethical "direction." So I would say you are throwing the bullion out with the bathwater. Money is a very efficient way of focusing effort - as your example indicates. How it is focused then is up to what we collectively value and how much we want to invest in determining where it will find value.
brudgers 13y ago
Larry Page's view that the last best hope for humanity is some white guy with the hubris to tell the residents of Camden that his not being allowed to sell $60,000 luxury cars is the most pressing issue they faced, didn't surprise me. The American notion of Manifest Destiny isn't dead. It's not even past. The mistake that many people of the developed world seem to make is to think that the Page's of the world see us as a distinct class apart from the billion global poor who live on less than $1 a day. It's a straight line from their suffering being just the business cost of sending rockets to Mars, to our $60,000 a year lives being the cost of doing it at scale. Right now, we can be comfortable in sharing Page's and Elop's abstractions. People we know aren't effected. When it comes our time, the best we can hope for is to be the Sixth Army on the Volga and get some lip service that our suffering was honorable and our death glorious and will go down in history a thousand years hence. Look up your street to the left, then down it to the right. If you don't see a native American, it's probably you.
- lnanek2 13y ago
  I do feel for people dying of Polio in third world countries. But it is just silly to think that Gates style philanthropy of saving those people is going to change the world, though. It just means a few more people in third world countries will live who might not, and those countries might perform a little better GDP wise and take care of their people a little bit better, and a microscopic few of those people may someday make a change in the world by inventing something or creating some huge company or something. Meanwhile, getting everyone in the US to use electric vehicles instead of gas would make a huge difference in the world. A society where transportation costs are much lower will be a very different society. Throw in Google self driving cars and we may not be mostly jammed into crime ridden cities any more, for example, since commuting would be a few pennies of electricity sitting in a cabin sleeping or working on a laptop instead of driving.
  - ianferrel 13y ago
    Gates-style philanthropy is on a course to eradicate polio. That's a potentially huge impact. Sure, the effects of polio are limited to a few poor people right now (mostly because herculean effort was already exerted to get it there), but if we can eradicate it, it means that not only is the current suffering eased, but we will never again have to spend any resources on polio, nor have people suffer from it.
gwern 13y ago
For the sake of argument, let's assume OP is correct: the finance industry and corporations in general are indeed blood-sucking vampire squids jamming their funnels into everything that smells of money and are antithetical to human values. (I strongly disagree with it, but others are criticizing that aspect, so I'll move on.) So what? How does this justify his conclusion: > But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure. Even if the finance industry / corporations are a parasite, so what? That doesn't wish away AI: AI will still remain a threat (or not a threat) regardless of whether Goldman Sachs is dissolved. Threats are not a zero-sum game: if you show that finance is a parasite, that doesn't suddenly alter the universe to guarantee AI is safe and easy. Even if we interpret him charitably as making a nuanced cost-effectiveness argument that 'we face many threats, have very limited resources, and corporations are more cost-effective to fight than working on AI', that still seems pretty darn unlikely. Corporations are a major political issue which attract the attention and effort of hundreds of thousands of people and inspire real world action like Occupy Wall Street; why does this fight need a few more people in it? Diminishing returns set in a very long time ago. And fundamentally, financial types have (or have not) been parasites throughout history. Their impact is bounded. They are not anything new under the sun. Whereas with AI, we don't know whether AI will ever exist; whether the impact will be huge or close to zero; hardly anyone takes it remotely seriously (quick question: have you ever read a mainstream article which did not make either a _Termintor_ or 'Rapture of the Nerds' allusion?); and the sum total of all efforts on thinking about the topic is, what, maybe $7m a year? (Google's safety board + MIRI + FHI + the new existential risk org whose name escapes me). You have to assume some pretty extreme values to make cost-effectiveness criterion spit out 'nobody should be thinking about AI! Everyone should be swotting up their Marx!' Conflating socially-harmful corporations with 'hostile AI' does both topics a disservice.
- ispolin 13y ago
  An even more charitable interpretation might be: We can gain insight on how to manage fully autonomous rule-based systems (AI) by looking at existing semi-autonomous rule-based systems (Corporations, Governments).
jerf 13y ago
It's a valid point on its own, but it's not a valid counterargument against the Singularity Institute. Human gatherings are still created from humans, which constrains the space of what they are, what they can do, and what they can think. Arguably they've evolved along with us, and what we have today are still logical extensions of the political structures we've been evolving for the last few million years. AIs have no such constraints. However alien a financial corporation may be, it might as well be a single human compared to what AIs could be. If the problem is real, it's a different problem, and the problem of large human corporations is only the smallest, smallest taste of the problem.
Hydraulix989 13y ago
I think there's other forms of hostile AI too like the algorithms that determine what advertisements to show in order to pollute your mind with intentionally-injected memes.
- thibauts 13y ago
  It's not pollution, only a market-share of your thoughts and actions.
- Florin_Andrei 13y ago
  I fully expected the essay to be on that topic.
saucetenuto 13y ago
Usually in these discussions we use the term "Unfriendly AI" instead of "Hostile AI". That gets at an important distinction: these AIs don't want humanity to die, it's just that they don't want humanity to live and they're presumed to have the ability to steal all our resources for themselves. Humanity still dies, but only incidentally. The author discusses this point a little but I think it's important enough to put it front and center, in our terminology. It's interesting to think of corporations as being Unfriendly in this sense. The analogy isn't perfect, though: humans make up a corporation's computing substrate, so they're forced to value humans more than the classic Unfriendly AI would.
- dnautics 13y ago
  is there a third category of AI which doesn't necessarily care to live or not, but winds up destroying humanity because of its Lennie-like power (possibly even in the service of humanity)?
  - saucetenuto 13y ago
    Maybe? I'm not sure what you mean.
    - eurleif 13y ago
      I took it to mean http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - maaku 13y ago
    Yes, "Unfriendly AI", as in not-provably-friendly AI, which is not the same thing as "Hostile AI".
sharemywin 13y ago
Financial institutions provide value. not sure how you buy a house without a mortgage. Not sure how you come back from your house burning down without insurance.
- 911_Inside_Job 13y ago
  Let me state this here and now, and I hope it's heard and recognized by you and by all others who take the "corpotations are GOOD not bad because we have so much wealth due to them, and we wouldn't have it without them" line. It is at the least mistaken to believe that. The angrier I get, the more likely I am to all call it wrong-headed and self-serving and then more foul things down from there. It's mistaken -- as may be perfectly characterized by "not sure how you buy a house without a mortgage!" -- to believe that you couldn't have a house without a financial corporation to loan you casino poker chips out of thin air (oops I mean loan you money on a fractional reserve basis) to buy it. And it analogously mistaken to think that corporations are sine-qua-non for the provision of comfort and needs. Our ways of doing business, lawful and entrenched and paradigmatic as they are, are just abstractions above the actual business of value creation and provision. They are not the only systems or abstract rules which may provide us the good things in life. Got that? The entrenched ways of providing things are not the only way. So far as my argument has come, I think logical minds should recognize that it's non-sequitur to think that the current paradigm of finance and business is the only way in which we could provision material comfort, and indeed "buy a house". Next, let me see if I can further convince you that what we have is actually BAD. (Having done away with the notion that this system is good AND necessary by establishing that it's not necessary, I want to further establish that it's not-good AND not-necessary.) It is the provision of money through debt which is just about the most fundamental mechanic that is (what I will call) BAD. I say this because I think it would be GOOD if we could build towards, and subsequently achieve, a healthy steady-state balance with our planet or planetary system. But the provision of money through debt and only debt sets up a world where there is always more debt than money, as the money owed in debt is always more than the money created by debt. Competition for this "scarce resource" money will keep humans on an eternal Sisyphean Hamster Wheel generation in and generation out as they can't EVER come to a healthy balance of economic production and consumption. We don't need entropy and heat-death to keep us busy working -- we've got a hunger for money to pay things off that's infinitely more ravenous than the fight against entropy. Got that? Money is created by debt issuance, and destroyed by debt repayment (that's right, destroyed by debt repayment just ask a fianncial accountant), and it takes more money to pay off any debt than the amount of money that was created by the debt. Ergo people can never achieve a balance and healthy steady state, as they have an imperative to expand and extract value (unsustainably extracting value) just to avoid bankruptcy. Whether it's balance with our planet, or planetary system, or beyond, as long as we keep having to go into debt to finance our houses and business expansions, it won't matter if the planet is twice as abundant in natural wealth, or if we expand to 10 wealthy planets, it will be flat-out impossible to come into macro balance between our artifical and abstracted "need" to pay back more money than we have and the real world in which we create value and luxury to earn said money. So, please, will you? Take your "hurr-hurr, durp-durp can't have a house without a mortgage" and throw it away and start thinking about the medium- and long-term results of our un-necessary and un-healthy system. Start thinking and talking in more realistic terms. Let's start provisioning our needs and comforts and luxuries without chaining ourselves to a broken paradigm. Seriously, we already have well more than half the people on the planet effectively barred from access to more than a very small pittance of the money in existence, separating the world into "first world" and "third world" societies, and yet the first world is struggling hard to not fall into collective bankruptcy and depression. We hog most of the money for ourselves, keep most of the other people out of wealth-buying casino chips (oops I mean money), and yet we in the E.U. and the U.S. still are struggling to make ends meet. That's our wonderful systems of finance and corporate organization for you. God forbid an A.I. should take them for granted, and actually believe it must compete for money.
davidgerard 13y ago
>All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure. This is one of my big problems with LessWrong (and I've been reading it four years): for all their claims of relevance, a startling unwillingness to question the social structure in a manner that would decrease the privileges of Bay Area techies like themselves. Politics is the mind-killer, don't you know. The offshoot Effective Altruist movement has the same problem: throw money at the symptoms (and denigrate anyone who doesn't do the same) while noticeably never, ever questioning the system the problems are in the context of.
dnautics 13y ago
"Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end." Well, no. Banks provide liquidity, investment opportunity, and loans. Just like McDonald's, there is a cost to this service, and the negatives of this cost may or may not be worth the "positive constraint" but to say there is no positive constraint is utterly rediculous. "Of all human institutions, these seem the most parasitical and dangerous." What of those institutions that wage wars and actually people killed?
- eitland 13y ago
  > What of those institutions that wage wars and actually people killed? keep in mind that these are the same institutions that at some point saved the rest of the European jews, the rest of the European gays, the rest of the European gypsies, the rest of the European communists, mentally or physically disabled etc.
  - bcoates 13y ago
    Nations: they haven't killed everyone
guscost 13y ago
A corporation or a government can appear to be autonomous in the same sense that a heroin addict can appear to be autonomous. Good luck convincing them to prioritize anything other than finding more heroin.
bachback 13y ago
DAC => https://en.bitcoin.it/wiki/Distributed_Autonomous_Community_/_Decentralized_Application https://en.bitcoin.it/wiki/Distributed_Autonomous_Community_...
webmaven 13y ago
cstross covered some of this same territory a while back (except that rather than 'unfriendly AIs', corporations are 'invading aliens'): http://www.antipope.org/charlie/blog-static/2010/12/invaders-from-mars.html http://www.antipope.org/charlie/blog-static/2010/12/invaders...
inetsee 13y ago
There is a link to a "book-length pdf" at "http://singularity.org/files/CFAI.pdf" http://singularity.org/files/CFAI.pdf", but the link shows up as a 404. I am curious about what the "book-length pdf" discusses, but my best efforts at finding it have failed. Anybody have an alternate link, or more information about where it might be found?
- Tenobrus 13y ago
  I'm pretty sure it's now http://intelligence.org/files/CFAI.pdf http://intelligence.org/files/CFAI.pdf. The organization in question changed its name and domain somewhat recently.
  - inetsee 13y ago
    Found it. Thanks!
  - mtraven 13y ago
    Fixed in post, thanks (and insert some juvenile snark about how-are-you-gonna-make-a-superintelligence-when-you-can't-even-keep-your-links-from-rotting? here).
    - gwern 13y ago
      (Redirects would be the responsibility of, and reflect on, the organization which bought the domain, not the one which sold it.)
TempleOSV2 13y ago
Why the fuck do you have such a problem accepting the Bible? Fucken niggers. You think they made that shit up? Study it closely. The psychology is obviously not bogus. Take cain and abel. ----- Adam[a] made love to his wife Eve, and she became pregnant and gave birth to Cain.[b] She said, “With the help of the Lord I have brought forth[c] a man.” 2 Later she gave birth to his brother Abel. Now Abel kept flocks, and Cain worked the soil. 3 In the course of time Cain brought some of the fruits of the soil as an offering to the Lord. 4 And Abel also brought an offering—fat portions from some of the firstborn of his flock. The Lord looked with favor on Abel and his offering, 5 but on Cain and his offering he did not look with favor. So Cain was very angry, and his face was downcast. 6 Then the Lord said to Cain, “Why are you angry? Why is your face downcast? 7 If you do what is right, will you not be accepted? But if you do not do what is right, sin is crouching at your door; it desires to have you, but you must rule over it.” 8 Now Cain said to his brother Abel, “Let’s go out to the field.”[d] While they were in the field, Cain attacked his brother Abel and killed him. 9 Then the Lord said to Cain, “Where is your brother Abel?” “I don’t know,” he replied. “Am I my brother’s keeper?” --------------- I can randomly open a book or randomly flip coins for random numbers. ---------- http://biblehub.com/proverbs/16-33.htm http://biblehub.com/proverbs/16-33.htm -------- Fucken niggers. Why can you not accept that centuries old wisdom has some basis in empiracle evidence. God says... what_luck you_better_not that's_for_me_to_know wrath fool hippy good nope spending cracks_me_up happy kludge Enough
codeulike 13y ago
Brings to mind Accelerando by Charles Stross, in which the baddies are post-singularity AI/corporation hybrids.
tptacek 13y ago
To the extent that this is true, it seems like it's been true for pretty much the whole of human history.
fu9ar 13y ago
I'm glad that all of those intelligent idiots are stuck writing HFT algorithms for a middling salary on Wall Street so that the rest of us who are playing a better game can get on with life. Money isn't everything.
deleted 13y ago
[deleted]
UK-AL 13y ago
Finance gets capital in the right place, at the right time. That provides value.
lifeisstillgood 13y ago
Which logical fallacy / form of argument is being raised here? It's a match for political issues (Say Scottish Independance - if we have AI / Independance, then we can do "improve human race / Create fairer society" - but then the argument goes why not just do the "fairer society thing" now as best you can perhaps it is the "What are you really waiting for?" argument (probably a favourable shift in power, to answer my own question)

130 points by mortenjorck 13y ago

A blog about the evil corporations, hosted by Google. I love the internet.

So McDonalds is ok because it is at least somewhat restrained by the need to offer a useful good (food) but the financial industry is parasitical and evil and offers nothing of use? Seriously?

evv 13y ago
I think you misunderstand. The author is saying that all corporations are "parasitical" and even "evil", for humans that is. McDonalds would do anything to squeeze more money out of its customers. Like most corporations, they profit from manipulating the human masses.
- zeno334 13y ago
  Uhm, whatever his other points may be, I can't take him seriously when he lumps all financial institutions together into one group that offers nothing of use. Sure, you can argue all day against the usefulness of speculative speed trading firms - and even then you'd still have some decent arguments against you - but to argue that financial institution in general offer nothing of use is ludicrous. Some relevant quotes: Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end. Of all human institutions, these seem the most parasitical and dangerous. it [the financial system] has interests that are distant or hostile to human goals
  - at-fates-hands 13y ago
    This is the textbook definition of utilitarianism: "Utilitarianism is a theory in normative ethics holding that the proper course of action is the one that maximizes utility, usually defined as maximizing happiness and reducing suffering." He sees financial institutions as a waste, and therefore consuming vast amounts of resources to no useful or positive effect.
  - fennecfoxen 13y ago
    Yeah... it's easy to criticize the financial network. So would you mind if we tacked on an extra N% to your mortgage payment, like you'd expect with less efficient financing? :)
norswap 13y ago
How would you say this assessment is wrong?
- aetherson 13y ago
  Well, it's wrong in no small part because the financial industry does, of course, produce something of value. I don't really get how the hard core "finance is fucked up" people imagine that financial firms got started. Do they imagine they were just created by evil aliens or something? Finance firms provide lots of very sophisticated ways for money to get from people who have lots of money that they don't currently have a use for, into the hands of people who have a need for money they don't currently have. Note this is a weak claim. I'm not saying anything about the efficiency of this process or whether the financial industry is good or evil or anything. But you can't criticize the finance industry without at least understanding and engaging its outputs.
  - Florin_Andrei 13y ago
    Computer, per se, are not bad. Computers infected with malware are bad. Infected computers can still do useful things for you - and in the background they'll be screwing you over also. The argument I would like to make is not that finance is bad intrinsically - as you've said above. The real problem is that the current instantiation of this concept has grown a series of goals of its own, and to fulfill those goals it is doing things that use the common, shared, limited pool of resources of the global economy, siphoning off stuff that the rest of us could put to better use. In other words, they could perform useful services for us, and they do. It's all the extra layer on top of that that's the problem. In a strict, mathematical sense, the OP probably exaggerated when saying they are entirely parasitical.
- zeno334 13y ago
  Uhm, well, the financial industry isn't parasitical and evil nor does it have interests that are distant or hostile to human goals. It's just another profit oriented industry that offers services which plenty of entities find valuable enough to pay for. The financial industry is as evil as any profit oriented industry can be and to single it out as something completely wasteful and parasitic is unreasonable.
effdee 13y ago
"The only useful banking innovation was the invention of the ATM." -- Paul Volcker
sp332 13y ago
McDonald's isn't "ok", but it's limited in how evil it can be because it has to give some benefit to other people. A hostile AI could be even worse, because it doesn't have that constraint.
- zeno334 13y ago
  But surely you can't compare a hostile AI which can do unlimited harm due to this lack of constraint to the financial industry can you?
  - sp332 13y ago
    Why not? A 2012 report estimated over 19 trillion dollars in lost household wealth, and contribution toward the largest number of people living in poverty ever recorded by the US Census bureau. http://www.pbs.org/wgbh/pages/frontline/business-economy-financial-crisis/money-power-wall-street/how-much-did-the-financial-crisis-cost/ http://www.pbs.org/wgbh/pages/frontline/business-economy-fin...
- randomdata 13y ago
  The financial system is what provides farmers with the necessary capital to be able to grow the food McDonalds needs to be in business. That doesn't necessarily make them okay either, but isn't allowing people to eat, without having to worry about producing their own food, at least of some benefit as well?
  - sp332 13y ago
    Financial institutions aren't inherently bad, but the ones we have are bad.
    - randomdata 13y ago
      Doesn't 'financial institution' cover a pretty wide gamut? There are small institutions that exist in only a small town, to government owned entities, to giant businesses that span the entire world. Are they all categorically bad? What can be done to make them less bad?

deleted 13y ago

[deleted]

NoodleIncident 13y ago

The jab at the financial industry is unnecessary, but the actual point is very interesting. I first heard of this as the 'Santa Claus' argument on a forum thread. Short version: just because you can't point at a single person and say 'he is Santa', doesn't mean that Santa doesn't exist. Despite living solely as a meme in people's heads, he has desires (make kids happy on Christmas, make kids behave), and can act on them (put presents under the tree). Those who supported this position used Microsoft as an example. 'Microsoft', like every company, is at its core an idea or a meme that everyone just accepts exists. Parts of it get written down, of course, and various people hold different parts of it in their head. Nevertheless, the company is its own entity, with its own goals and means of achieving them, yet exists only in people's heads.

pdonis 13y ago
But since Santa only exists as a meme in people's heads, Santa can't do anything without the people in whose heads the meme exists doing something. Similarly, corporations can't do anything without the people that own them and work for them doing something. So viewing corporations as quasi-autonomous agents is misleading; they can be viewed as collective agents in some respects, but in the end everything they do has to come down to some human or humans taking some actions. Basically, the author of this article is punting: he's saying the problem is corporations being "effectively independent of human control", instead of actually looking at the actual humans whose decisions and actions constitute what the corporations do.
- sutterbomb 13y ago
  I don't think he's punting. He's saying that actually looking at the actual humans is misguided, because they can and will be replaced or exchanged with no effect on the system as a whole. At the micro level, a department or even a company may change course in a noticeable way with one person being replaced. But industry wide, and society-wide, the "actual people" are mostly irrelevant to the mechanics of the system.
  - pdonis 13y ago

yukichan 13y ago

If corporations are so evil why has their pay fed my family, paid for my house and created so many amazing products we surround ourselves with. It's ridiculous. Big organizations have problems, but to pronounce them as inherently evil ignores all the good they do. Maybe you have a different idea of good in mind, in which case we'll probably never agree on anything. When people organize they can create amazing things, and that's what corporations are, an organization of people: employees, customers, and investors. And they do amazing things. Go to a country that doesn't do so well at creating functioning organizations and see the difference.

r00fus 13y ago
Let's not dilute the point. The article states that the financial industry* is the "hostile AI" not all corporations (in fact, it clearly states that even the hated "energy" corporations actually provide a product - even if that ). Having recently re-read Dan Simmons' Hyperion series (sci-fi humans vs. AI masterpiece), I could almost see the analogy perfectly - finance is key to almost every part of our individual life, business and even political machinations. And in each, the big finance outfits are impervious, indifferent and have their own goals outside of their stated purpose. * I'd expand slightly to include all of FIRE - finance, real-estate and insurance
dropit_sphere 13y ago
They're not evil, they're not good. They just are. Blind worship and blind fear are errors of the same category: mistaking their nature.
ThomPete 13y ago
I am not disagreeing fundamentally but you can't use your success as an indicator of whether something is good or bad. The corporation that feed you might do other things other places to other people. As an example the De Beers in the diamond industry.
harrystone 13y ago
It's just much easier to think of things as being black and white. It really doesn't matter if it's true, it's just comforting.

fennecfoxen 13y ago

Naah, corporations aren't independent profit-seeking AIs. They don't bother making money except insofar as their owners drive them to. Even in that state, they're subject to principal/agent conflicts in which people seek to turn the company's revenue into their own overpriced salaries / stock options. Now, government bureaucracies may be another story, especially once they grow lobbying arms ;)

madaxe_again 13y ago

The point here is that the generation of wealth, which is the conversion of labour and marginal efficiency into a number in a ledger, is not in and of itself aligned with human interests - it turns valuable labour and intelligence into a store of labour-time exchange, which can be used to purchase more labour, rather than any absolute benefit. The problem arises when the store of capital grossly exceeds available useful labour, which it does now in the hands of those who hold it. This causes inflationary forces, the results of which are very visible, and in this way the excess labour-time is simply destroyed, as it becomes devalued. Money is a highly inefficient way of improving the world.

AndrewKemendo 13y ago
>Money is a highly inefficient way of improving the world. True vis a vis itself - as money in and of itself does not have an ethical "direction." So I would say you are throwing the bullion out with the bathwater. Money is a very efficient way of focusing effort - as your example indicates. How it is focused then is up to what we collectively value and how much we want to invest in determining where it will find value.

brudgers 13y ago

Larry Page's view that the last best hope for humanity is some white guy with the hubris to tell the residents of Camden that his not being allowed to sell $60,000 luxury cars is the most pressing issue they faced, didn't surprise me. The American notion of Manifest Destiny isn't dead. It's not even past. The mistake that many people of the developed world seem to make is to think that the Page's of the world see us as a distinct class apart from the billion global poor who live on less than $1 a day. It's a straight line from their suffering being just the business cost of sending rockets to Mars, to our $60,000 a year lives being the cost of doing it at scale. Right now, we can be comfortable in sharing Page's and Elop's abstractions. People we know aren't effected. When it comes our time, the best we can hope for is to be the Sixth Army on the Volga and get some lip service that our suffering was honorable and our death glorious and will go down in history a thousand years hence. Look up your street to the left, then down it to the right. If you don't see a native American, it's probably you.

lnanek2 13y ago
I do feel for people dying of Polio in third world countries. But it is just silly to think that Gates style philanthropy of saving those people is going to change the world, though. It just means a few more people in third world countries will live who might not, and those countries might perform a little better GDP wise and take care of their people a little bit better, and a microscopic few of those people may someday make a change in the world by inventing something or creating some huge company or something. Meanwhile, getting everyone in the US to use electric vehicles instead of gas would make a huge difference in the world. A society where transportation costs are much lower will be a very different society. Throw in Google self driving cars and we may not be mostly jammed into crime ridden cities any more, for example, since commuting would be a few pennies of electricity sitting in a cabin sleeping or working on a laptop instead of driving.
- ianferrel 13y ago
  Gates-style philanthropy is on a course to eradicate polio. That's a potentially huge impact. Sure, the effects of polio are limited to a few poor people right now (mostly because herculean effort was already exerted to get it there), but if we can eradicate it, it means that not only is the current suffering eased, but we will never again have to spend any resources on polio, nor have people suffer from it.

gwern 13y ago

For the sake of argument, let's assume OP is correct: the finance industry and corporations in general are indeed blood-sucking vampire squids jamming their funnels into everything that smells of money and are antithetical to human values. (I strongly disagree with it, but others are criticizing that aspect, so I'll move on.) So what? How does this justify his conclusion: > But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure. Even if the finance industry / corporations are a parasite, so what? That doesn't wish away AI: AI will still remain a threat (or not a threat) regardless of whether Goldman Sachs is dissolved. Threats are not a zero-sum game: if you show that finance is a parasite, that doesn't suddenly alter the universe to guarantee AI is safe and easy. Even if we interpret him charitably as making a nuanced cost-effectiveness argument that 'we face many threats, have very limited resources, and corporations are more cost-effective to fight than working on AI', that still seems pretty darn unlikely. Corporations are a major political issue which attract the attention and effort of hundreds of thousands of people and inspire real world action like Occupy Wall Street; why does this fight need a few more people in it? Diminishing returns set in a very long time ago. And fundamentally, financial types have (or have not) been parasites throughout history. Their impact is bounded. They are not anything new under the sun. Whereas with AI, we don't know whether AI will ever exist; whether the impact will be huge or close to zero; hardly anyone takes it remotely seriously (quick question: have you ever read a mainstream article which did not make either a _Termintor_ or 'Rapture of the Nerds' allusion?); and the sum total of all efforts on thinking about the topic is, what, maybe $7m a year? (Google's safety board + MIRI + FHI + the new existential risk org whose name escapes me). You have to assume some pretty extreme values to make cost-effectiveness criterion spit out 'nobody should be thinking about AI! Everyone should be swotting up their Marx!' Conflating socially-harmful corporations with 'hostile AI' does both topics a disservice.

ispolin 13y ago
An even more charitable interpretation might be: We can gain insight on how to manage fully autonomous rule-based systems (AI) by looking at existing semi-autonomous rule-based systems (Corporations, Governments).

jerf 13y ago

It's a valid point on its own, but it's not a valid counterargument against the Singularity Institute. Human gatherings are still created from humans, which constrains the space of what they are, what they can do, and what they can think. Arguably they've evolved along with us, and what we have today are still logical extensions of the political structures we've been evolving for the last few million years. AIs have no such constraints. However alien a financial corporation may be, it might as well be a single human compared to what AIs could be. If the problem is real, it's a different problem, and the problem of large human corporations is only the smallest, smallest taste of the problem.

Hydraulix989 13y ago

I think there's other forms of hostile AI too like the algorithms that determine what advertisements to show in order to pollute your mind with intentionally-injected memes.

thibauts 13y ago
It's not pollution, only a market-share of your thoughts and actions.
Florin_Andrei 13y ago
I fully expected the essay to be on that topic.

saucetenuto 13y ago

Usually in these discussions we use the term "Unfriendly AI" instead of "Hostile AI". That gets at an important distinction: these AIs don't want humanity to die, it's just that they don't want humanity to live and they're presumed to have the ability to steal all our resources for themselves. Humanity still dies, but only incidentally. The author discusses this point a little but I think it's important enough to put it front and center, in our terminology. It's interesting to think of corporations as being Unfriendly in this sense. The analogy isn't perfect, though: humans make up a corporation's computing substrate, so they're forced to value humans more than the classic Unfriendly AI would.

dnautics 13y ago
is there a third category of AI which doesn't necessarily care to live or not, but winds up destroying humanity because of its Lennie-like power (possibly even in the service of humanity)?
- saucetenuto 13y ago
  Maybe? I'm not sure what you mean.
  - eurleif 13y ago
    I took it to mean http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
- maaku 13y ago
  Yes, "Unfriendly AI", as in not-provably-friendly AI, which is not the same thing as "Hostile AI".

sharemywin 13y ago

Financial institutions provide value. not sure how you buy a house without a mortgage. Not sure how you come back from your house burning down without insurance.

911_Inside_Job 13y ago
Let me state this here and now, and I hope it's heard and recognized by you and by all others who take the "corpotations are GOOD not bad because we have so much wealth due to them, and we wouldn't have it without them" line. It is at the least mistaken to believe that. The angrier I get, the more likely I am to all call it wrong-headed and self-serving and then more foul things down from there. It's mistaken -- as may be perfectly characterized by "not sure how you buy a house without a mortgage!" -- to believe that you couldn't have a house without a financial corporation to loan you casino poker chips out of thin air (oops I mean loan you money on a fractional reserve basis) to buy it. And it analogously mistaken to think that corporations are sine-qua-non for the provision of comfort and needs. Our ways of doing business, lawful and entrenched and paradigmatic as they are, are just abstractions above the actual business of value creation and provision. They are not the only systems or abstract rules which may provide us the good things in life. Got that? The entrenched ways of providing things are not the only way. So far as my argument has come, I think logical minds should recognize that it's non-sequitur to think that the current paradigm of finance and business is the only way in which we could provision material comfort, and indeed "buy a house". Next, let me see if I can further convince you that what we have is actually BAD. (Having done away with the notion that this system is good AND necessary by establishing that it's not necessary, I want to further establish that it's not-good AND not-necessary.) It is the provision of money through debt which is just about the most fundamental mechanic that is (what I will call) BAD. I say this because I think it would be GOOD if we could build towards, and subsequently achieve, a healthy steady-state balance with our planet or planetary system. But the provision of money through debt and only debt sets up a world where there is always more debt than money, as the money owed in debt is always more than the money created by debt. Competition for this "scarce resource" money will keep humans on an eternal Sisyphean Hamster Wheel generation in and generation out as they can't EVER come to a healthy balance of economic production and consumption. We don't need entropy and heat-death to keep us busy working -- we've got a hunger for money to pay things off that's infinitely more ravenous than the fight against entropy. Got that? Money is created by debt issuance, and destroyed by debt repayment (that's right, destroyed by debt repayment just ask a fianncial accountant), and it takes more money to pay off any debt than the amount of money that was created by the debt. Ergo people can never achieve a balance and healthy steady state, as they have an imperative to expand and extract value (unsustainably extracting value) just to avoid bankruptcy. Whether it's balance with our planet, or planetary system, or beyond, as long as we keep having to go into debt to finance our houses and business expansions, it won't matter if the planet is twice as abundant in natural wealth, or if we expand to 10 wealthy planets, it will be flat-out impossible to come into macro balance between our artifical and abstracted "need" to pay back more money than we have and the real world in which we create value and luxury to earn said money. So, please, will you? Take your "hurr-hurr, durp-durp can't have a house without a mortgage" and throw it away and start thinking about the medium- and long-term results of our un-necessary and un-healthy system. Start thinking and talking in more realistic terms. Let's start provisioning our needs and comforts and luxuries without chaining ourselves to a broken paradigm. Seriously, we already have well more than half the people on the planet effectively barred from access to more than a very small pittance of the money in existence, separating the world into "first world" and "third world" societies, and yet the first world is struggling hard to not fall into collective bankruptcy and depression. We hog most of the money for ourselves, keep most of the other people out of wealth-buying casino chips (oops I mean money), and yet we in the E.U. and the U.S. still are struggling to make ends meet. That's our wonderful systems of finance and corporate organization for you. God forbid an A.I. should take them for granted, and actually believe it must compete for money.

davidgerard 13y ago

>All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure. This is one of my big problems with LessWrong (and I've been reading it four years): for all their claims of relevance, a startling unwillingness to question the social structure in a manner that would decrease the privileges of Bay Area techies like themselves. Politics is the mind-killer, don't you know. The offshoot Effective Altruist movement has the same problem: throw money at the symptoms (and denigrate anyone who doesn't do the same) while noticeably never, ever questioning the system the problems are in the context of.

dnautics 13y ago

"Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end." Well, no. Banks provide liquidity, investment opportunity, and loans. Just like McDonald's, there is a cost to this service, and the negatives of this cost may or may not be worth the "positive constraint" but to say there is no positive constraint is utterly rediculous. "Of all human institutions, these seem the most parasitical and dangerous." What of those institutions that wage wars and actually people killed?

eitland 13y ago
> What of those institutions that wage wars and actually people killed? keep in mind that these are the same institutions that at some point saved the rest of the European jews, the rest of the European gays, the rest of the European gypsies, the rest of the European communists, mentally or physically disabled etc.
- bcoates 13y ago
  Nations: they haven't killed everyone

guscost 13y ago

A corporation or a government can appear to be autonomous in the same sense that a heroin addict can appear to be autonomous. Good luck convincing them to prioritize anything other than finding more heroin.

bachback 13y ago

DAC => https://en.bitcoin.it/wiki/Distributed_Autonomous_Community_/_Decentralized_Application https://en.bitcoin.it/wiki/Distributed_Autonomous_Community_...

webmaven 13y ago

cstross covered some of this same territory a while back (except that rather than 'unfriendly AIs', corporations are 'invading aliens'): http://www.antipope.org/charlie/blog-static/2010/12/invaders-from-mars.html http://www.antipope.org/charlie/blog-static/2010/12/invaders...

inetsee 13y ago

There is a link to a "book-length pdf" at "http://singularity.org/files/CFAI.pdf" http://singularity.org/files/CFAI.pdf", but the link shows up as a 404. I am curious about what the "book-length pdf" discusses, but my best efforts at finding it have failed. Anybody have an alternate link, or more information about where it might be found?

Tenobrus 13y ago
I'm pretty sure it's now http://intelligence.org/files/CFAI.pdf http://intelligence.org/files/CFAI.pdf. The organization in question changed its name and domain somewhat recently.
- inetsee 13y ago
  Found it. Thanks!
- mtraven 13y ago
  Fixed in post, thanks (and insert some juvenile snark about how-are-you-gonna-make-a-superintelligence-when-you-can't-even-keep-your-links-from-rotting? here).
  - gwern 13y ago
    (Redirects would be the responsibility of, and reflect on, the organization which bought the domain, not the one which sold it.)

TempleOSV2 13y ago

Why the fuck do you have such a problem accepting the Bible? Fucken niggers. You think they made that shit up? Study it closely. The psychology is obviously not bogus. Take cain and abel. ----- Adam[a] made love to his wife Eve, and she became pregnant and gave birth to Cain.[b] She said, “With the help of the Lord I have brought forth[c] a man.” 2 Later she gave birth to his brother Abel. Now Abel kept flocks, and Cain worked the soil. 3 In the course of time Cain brought some of the fruits of the soil as an offering to the Lord. 4 And Abel also brought an offering—fat portions from some of the firstborn of his flock. The Lord looked with favor on Abel and his offering, 5 but on Cain and his offering he did not look with favor. So Cain was very angry, and his face was downcast. 6 Then the Lord said to Cain, “Why are you angry? Why is your face downcast? 7 If you do what is right, will you not be accepted? But if you do not do what is right, sin is crouching at your door; it desires to have you, but you must rule over it.” 8 Now Cain said to his brother Abel, “Let’s go out to the field.”[d] While they were in the field, Cain attacked his brother Abel and killed him. 9 Then the Lord said to Cain, “Where is your brother Abel?” “I don’t know,” he replied. “Am I my brother’s keeper?” --------------- I can randomly open a book or randomly flip coins for random numbers. ---------- http://biblehub.com/proverbs/16-33.htm http://biblehub.com/proverbs/16-33.htm -------- Fucken niggers. Why can you not accept that centuries old wisdom has some basis in empiracle evidence. God says... what_luck you_better_not that's_for_me_to_know wrath fool hippy good nope spending cracks_me_up happy kludge Enough

codeulike 13y ago

Brings to mind Accelerando by Charles Stross, in which the baddies are post-singularity AI/corporation hybrids.

tptacek 13y ago

To the extent that this is true, it seems like it's been true for pretty much the whole of human history.

fu9ar 13y ago

I'm glad that all of those intelligent idiots are stuck writing HFT algorithms for a middling salary on Wall Street so that the rest of us who are playing a better game can get on with life. Money isn't everything.

deleted 13y ago

[deleted]

UK-AL 13y ago

Finance gets capital in the right place, at the right time. That provides value.

lifeisstillgood 13y ago

Which logical fallacy / form of argument is being raised here? It's a match for political issues (Say Scottish Independance - if we have AI / Independance, then we can do "improve human race / Create fairer society" - but then the argument goes why not just do the "fairer society thing" now as best you can perhaps it is the "What are you really waiting for?" argument (probably a favourable shift in power, to answer my own question)