18 ms·

Should AI Be Open?

• beat 11y ago
  Should AI be open? Depends on whether the AI is capable of deciding for itself whether it should be open or not.
• mindcrime 11y ago
  So, here's a random thought on this whole subject of "AI risk". Bostrom, Yudkowsky, etc. posit that an "artificial super-intelligence" will be many times smarter than humans, and will represent a threat somewhat analogous to an atomic weapon. BUT... consider that the phrase "many times smarter than humans" may not even mean anything. Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. Or close enough that being "smarter than human" does not represent anything analogous to an atomic bomb. So this might be an interesting topic for research, or at least for the philosophers: "What's the limit of how 'smart' it's possible to be"? It may be that there's no possible way to determine that (you don't know what you don't know and all that) but if there is, it might be enlightening.
  • Moshe_Silnorin 11y ago
    >It's possible that we're already roughly as intelligent as it's possible to be. Now that's more depressing than global annihilation.
    • mindcrime 11y ago
      > Now that's more depressing than global annihilation. Fair point. But I honestly wonder if it's not true, or close to true. I mean, people have been talking about the "end of science" for a while now, even though we know we're not quite literally at "the end". But let's say that we unify relativity / gravity / QM in the next 100 years or so, and identify the exact nature of dark energy / dark matter. Presumably that would represent knowing close to everything about the physical world that its possible to know. And if we can discover that, it does lead you to wonder "what else would more intelligence represent"? Of course, maybe we're WAY more than 100 years away from those things. Or maybe we'll never get there. Or maybe if we do, it doesn't mean anything vis-a-vis the limits of intelligence. I'm just thinking out loud here, this isn't an argument that I've put a lot of time into developing...
      • Kutta 11y ago
        > would represent knowing close to everything about the physical world that its possible to know. That's a severe misunderstanding of what it means to have a unified physical theory. The current standard model of physics allows us to theoretically predict the evolution of physical systems with an insanely high precision. A precision so great that there wouldn't be any meaningful error simulating macroscopic objects like human brains, given sufficient computing power. But we still have massive holes in our understanding of many things, including human biology or neurology. We don't have enough computing power to precisely simulate, but even if we had that power, it wouldn't necessarily let us understand anything. Elementary particles and humans are on hugely different scales. Observing a simulated human would be easier than observing a human simulated by physics itself, but it wouldn't just magically make everything clear. Having a TOE in physics brings us very little relative benefit compared to the Standard Model, when it comes to understanding biology, materials science, geology, psychology, computer science, etc.
        • mindcrime 11y ago
          That's a severe misunderstanding of what it means to have a unified physical theory. The current standard model of physics allows us to theoretically predict the evolution of physical systems with an insanely high precision. A precision so great that there wouldn't be any meaningful error simulating macroscopic objects like human brains, given sufficient computing power. Sure, like I said, I'm just thinking out loud here. Obviously TOE in and of itself doesn't automatically translate into full knowledge of everything in practice. But in principle, it would given sufficient computing power, allow us to simulate anything, which I think would yield additional understanding. But here's the thing, and where this all ties back together: sufficient computing resources may not be possible, even in principle, to allow use of that TOE to do $X. But an upper bound on the amount of computing resources should also reflect something of an upper bound on what our hypothetical AI can do as well. Or to put it all slightly differently.. if we had a TOE, we'd have shown, at least, that humans are smart enough to develop a TOE. Which I think at least raises the issue of "how much smarter can a machine be, or what would it mean for a machine to be much smarter than that?" Note to that I'm not necessarily arguing for this position. It's more of a thought experiment or discussion point than something I'm firmly convinced of.
    • AnimalMuppet 11y ago
      If you can't be more than you are now (or make something more), you'd rather not exist? That seems rather petulant of you.
      • Moshe_Silnorin 11y ago
        Lacking in principle the tools to solve the problems the universe gives you may be more depressing than non-existence. Though, I don't think it's actually possible to not exist. Name one person who's experienced nonexistence.
        • AnimalMuppet 11y ago
          I'm not sure that "experiencing nonexistence" is theoretically possible. However, I can name lots of people who existed, and now don't. (Even if you're a theist or believe in reincarnation, I'm talking about this current, earthly existence.)
    • isolate 11y ago
      I don't think a lack of intelligence of the IQ test variety is what's holding humanity back. I think it's distraction, greed, lack of empathy (or emotional intelligence), lack of information, and lack of communication. (Maybe there are more things.) Basically, I believe we have all the technology and intelligence we need to make this planet a much better place.
      • steve19 11y ago
        We had all the technology and intelligence we needed 2,000 years ago to make the world a better place. Yet our ancestors did not. For many 1000's of years different religions, some on opposite sides of the world from each other, have been saying the same thing. We have the ability to be better people, but all of us squander it with pettiness, jealousy, greed etc.
  • mrob 11y ago
    Even if that's true, imagine an AI as smart as John von Neumann on modafinil, that never thinks about food/sex/art/etc., that never sleeps, that has access to Wikipedia etc. at the speed of thought, and no morals. That's not uncontrolled intelligence explosion level disaster, but it's still highly dangerous.
    • murbard2 11y ago
      A virtual campus with thousands of hyperfocused John Von Neumanns collaborating telepathically and accomplish subjective years of works in seconds of real world time.
      • vectorjohn 11y ago
        The assumption being that these John Von Neumanns will only require cheap, modest hardware to run on.
        • deleted 11y ago
          [deleted]
        • cousin_it 11y ago
          If we have a Von Neumann level intelligence running on a supercomputer, and we haven't solved friendliness by then, we've lost the future of humanity. As far as I can tell, all arguments against that conclusion are based on various kinds of wishful thinking.
          • murbard2 11y ago
            Yo, dawg, I heard ya like Von Neumann architecture.
    • vectorjohn 11y ago
      And still I doubt that AI could talk its way out of a prison. It could be dangerous though.
  • Kutta 11y ago
    Even if human intelligence was the pinnacle, AI could be still extremely dangerous just by running at accelerated simulation speed and using huge amounts of subjective time to invent faster hardware. See https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf for discussion. The point is moot anyway though, since the hypothesis (that humans are the most intelligent possible) is just severely incompatible with our current understanding of science.
    • vectorjohn 11y ago
      This comes with the assumption that the first AI will be able to run at better than human levels on something like a raspbery pi. More realistically, it's going to have to run on an immense super computer, and even if it can marginally improve (marginal because a human level AI is no more likely to improve its own hardware than I am), it won't be able to just spread all over the world. It needs real physical hardware. That is, unless the reason we don't have AI is because we haven't put the pieces together correctly, and it really could work on minimal hardware.
      • smegger001 11y ago
        Just because it requires real hardware does not mean it may be able to spread all over the world. It may just do it by setting up virtual machines in data centers all over the world, or as a botnet with a distrusted processing approach similar to folding at home.
  • ctl 11y ago
    Well, if it's possible to build a human level intelligence, it's probably possible to build an intelligence that's much like a very smart human except it runs 100x faster. And in that case, somebody with sufficient resources could build an ensemble of 1000 superfast intelligences. That's a lower bound on the scariness of AI explosion, and it may already be enough to take over the world. Certainly it should be enough to take over the Internet circa 2015... To my mind it seems pretty clear that if AI exists, then scary AI is not far off. That said, I don't worry about this stuff too much, because I see AI as being much technically harder, and much less likely to materialize in our lifetimes, than articles like this suppose.
  • MBlume 11y ago
    If you look at the history of human evolution, this doesn't make sense. Evolution was very very slowly increasing human intelligence by, e.g., making our skulls bigger. Then we got to the point where we could use language/transmit knowledge between generations/etc. and got up to technological, spacefaring civilization in, evolutionarily speaking, no time whatsoever. This is not a story which suggests that human intelligence is some sort of maximum, or that evolution was running into diminishing returns and so stopped at human intelligence. It suggests that human intelligence is the minimum intelligence necessary to produce the kind of generational transfer that gets you up to technological civilization.
    • alextgordon 11y ago
      It's not like humans sit around all day thinking. We build tools to do the thinking for us. Humans ourselves are very bad at predicting the weather. So we build supercomputers, and weather models, and now we're a bit better at it. Now the question is: given the same budget, would a super-intelligent AI predict the weather in a substantially different way to humans? I cannot see how, but maybe that is my human stupidity talking.
      • Houshalter 11y ago
        This is a pretty random domain to pick. We don't know whether it's even possible to predict the weather better than we are. There are limits to that kind of prediction because weather is a chaotic system. However if it is possible to improve weather simulations, and if there are advancements to be made in meteorology, then it's very likely an AI would be able to find them.
      • lmm 11y ago
        Humans are very bad at self improving. Imagine how smart we'd be now if our intelligence had increased at the same rate that computing power increases. Now imagine how fast computing power would've increased if we'd been that smart. Now imagine...
  • tachyonbeam 11y ago
    Close to the limit of how smart it's possible to be? Don't be silly. The human brain is limited by its slow speed, by the amount of cortical mass you can fit inside a human skull, and by the length of human lifetimes. Computers will not have any of those limitations. In terms of speed: if you could build the exact silicon equivalent of a human brain, you may be able to run it several orders of magnitude faster, simply because it wouldn't be limited by the slow speeds of electrochemical processes in the human brain. Nerve impulses travel at speeds measured in meters per second. Neurons also need time to recharge between spike bursts, and they can physically damage themselves if they get too excited. In terms of volume: much of our intelligence is in perceiving patterns. That's limited by cortical mass. Pattern recognition is what all these "deep learning" systems excel at. The more depth they add, the better they get. Having deeper pattern recognizers, or simply having more of them, means you can see more patterns, more complex patterns, etc. Things that might be beyond the reach of any human. Then, in terms of data, machine have an advantage too. We're limited by our short lifetimes. How many people are expert musicians, genius mathematicians, rockstar programmers and great cooks? Very few. There's only so many hours in a day, and we only live so long. A machine could learn all those skills, and more. It could excel at everything. Speak every language, master every skill, be aware of so many more facts. And finally, I posit that maybe we, humans, are limited in our ability to grasp complex conceptual relationships. If you think about it, the average person can fit 7-8 items in their short-term memory, in their brain's "registers", so to speak. That probably limits our ability to reason by analogy. We can go "A is to B what C is to D", but maybe more complex relationships with 50 variables and several logical connectives will seem "intuitive" to a machine that can manipulate 200 items in its short-term memory.
    • mindcrime 11y ago
      The human brain is limited by its slow speed, by the amount of cortical mass you can fit inside a human skull, and by the length of human lifetimes. Computers will not have any of those limitations. Right, but the idea I'm playing around with is this: Suppose you had a hypothetical creature with a brain 20x as fast as the human brain and with twice the volume. How much smarter would that creature be in practice. It's kind of an abstract idea (and I probably don't fully understand it myself), but I'm getting at something like "is there a point where that additional raw computing power just doesn't buy you anything meaningful" at least in terms of "does it represent an existential threat?" or "does the nuclear analogy hold?"
  • skybrian 11y ago
    Alternately, perhaps it is possible to be much smarter, but it's not as effective as we expect? If we think of intelligence as skill at solving problems, it might be that there are not many problems that are easily solved with increased intelligence alone, because the solutions we have are nearly optimal, or because other inputs are needed as well. This seems most likely to happen with mathematical proofs and scientific laws. Increased intelligence doesn't let you prove something that's false, and it doesn't let you violate scientific laws. But I don't find this particularly plausible. Consider computer security: hackers are finding new exploits all the time. We're far from fixing all the loopholes that could possibly be found.
    • mindcrime 11y ago
      Alternately, perhaps it is possible to be much smarter, but it's not as effective as we expect? If we think of intelligence as skill at solving problems, it might be that there are not many problems that are easily solved with increased intelligence alone, because the solutions we have are nearly optimal, or because other inputs are needed as well. Yeah, I think that's a much better way of putting what I'm trying to get at. That is, maybe the machine is much "smarter" depending on how you define "smart". But maybe that doesn't enable it to do much more than we can do, because of other fundamental limits.
    • AJ007 11y ago
      That was the way I have been looking at it. It is not just an issue of finding solutions to impossible problems, or breaking scientific laws, but also problems where the solutions are along the lines of what George Soros would call reflexive. Computer security is like this, so is securities trading (no pun.) Secondly, what about problems which require the destruction of the problem solvers along the path to the optimal solution? I'm not sure about the correct word to describe this, or the best example but it could be seen in large systems. Where humans are right now is a result of this. We would not know many things if those things which came before were not destroyed (cities, 1000 year empires, etc.) Thirdly, is a uniform, singular AI the most optimal agent to solve these sorts of problems? Much the way we don't rely or use mainframes for computing today, perhaps there will be many AI agents each which may be really good at solving particular narrowly defined problem sets. This could be described perhaps as a swarm AI. Nick Bostrom's Superintelligence is a great book, but I don't recall much consideration along these lines. When a lot of AI agents are "live" the paths to solutions where AI compete against each other open up even more complex scenarios. There certainly are physical limitations to AI. Things like the speed of light can slow down processing. Consumption of energy. Physical elements that can be assembled for computational purposes. Between now and "super" AI, even really good AI could struggle to find solutions to the most difficult problems, especially if those are problems other AI are creating. The speed alone may be the largest challenge to humans. How do we measure this difficulty relative to human capabilities, I don't know. End of rant -- but the limits of not just AI but problem solving is quite interesting.
  • ggreer 11y ago
    > Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. I think Nick Bostrom had the perfect reply to that in Superintelligence: Paths, Dangers, Strategies: > Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. It would be extremely strange if we were near the smartest possible minds. Just look at the evidence: Our fastest neurons send signals at 0.0000004c. Our working memory is smaller than a chimp's.[1] We need pencil and paper to do basic arithmetic. These are not attributes of the pinnacle of possible intelligences. Even if you think it's likely that we are near the smartest possible minds, consider the consequences of being wrong: The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk. 1. https://www.youtube.com/watch?v=zsXP8qeFF6A https://www.youtube.com/watch?v=zsXP8qeFF6A
    • mindcrime 11y ago
      The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk. I acknowledge it as a real risk, but it's not terribly high on my personal list of things to worry about right now. I like what Andrew Ng said about how worrying about this now is "like worrying about over-population on Mars".
      • ggreer 11y ago
        Please notice that your reply is a different argument than the one you first put forth. Originally, you weren't worried about AI because you thought it could never –even in principle– vastly exceed human abilities. Now you're basically saying, "I don't need to worry because it won't happen for a long time." That is a huge amount of ground to cede. I'm not so confident that human-level AI will take a long time. The timeline depends on algorithmic insights, which are notoriously difficult to predict. It could be a century. It could be a decade. Still, it seems like something worth worrying about.
  • eli_gottlieb 11y ago
    > Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. No, there's clearly at least one human smarter than you.
    • mindcrime 11y ago
      Wait, what? I'm sure there are a lot of humans smarter than me. I'm talking about the overall approximate intelligence of humans in a general sense, not any specific individual.
      • eli_gottlieb 11y ago
        I certainly think there are individuals within our species whose cognitive algorithms are performing near-optimally. I just also have very good reason to think I can develop some algorithms that are better for certain tasks than the ones I consist in: I think there's sometimes more information in our sensory data than we take advantage of, and I think that better algorithms could tie together sensing and decision-making to obtain better data more often. Our brains "bite the Bayesian bullet" a bit too hard, accepting noisy data as-is and just making whatever inferences are feasible, rather than computing how to obtain clearer, less-noisy data (which is why we developed science as a form of domain knowledge rather than as an intuitive algorithm).
  • sawwit 11y ago
    I think one of the best evidences we have is the level at which computers outperform the human mind in certain domains. An artificial general intelligence would have very low latency access to all these mathematical and computational tools we've invented (physical simulations, databases, theorem provers), and it would not need to mechanically enter program code on a keyboard, but it would be directly wired to the compiler. It could possibly learn to think in program code and execute it on the fly. The computation environment of neurons is also extremely noisy (axons are not well insulated) and neurons only fire at 7-200Hz. Assuming noise and low firing rate do not fulfill a certain task in mammalian brains, this is another way in which silicon-based minds could potentially be vastly superior. Thirdly, assuming sleep is not necessary for intelligence, artificial minds would never get exhausted. They could work 24 hours on a problem a day, which is possibly 5-10 time the amount of thinking time a human can do realistically. And lastly, an AI could easily make copies of itself. Doing so it could branch a certain problem to many computers which run copies of it and eventually collect the best result, or just shorten the time it takes to get a result. It could also evolve at a much faster rate than humans, assuming it has a genetic description: possibly hours to seconds instead of 20 years. Anyhow, it could easily perform experiments with slightly changed version of itself.
  • Retra 11y ago
    I think the more relevant fact is that we don't have any ethical objections to shutting down computers, they're wholly dependent on our infrastructure, and the'll only evolve in ways that prove useful to us, because we wouldn't put a computer in charge of everything unless it were sufficiently compliant to our desires. I mean, are you going to put the same machine in charge of mineral extraction, weapon construction, transportation, and weapon deployment? When it hasn't proven to act correctly in a high-fidelity simulated environment? Probably not. We're also assuming that human ethics and intelligence are independent. I don't see many reasons to believe this. Social power and intelligence might be independent.
    • je42 11y ago
      it doesn't work like that. you get into trouble when there is a trade-off to be made between switching off the AI and leaving it on.
  • vectorjohn 11y ago
    I think most people didn't really understand the meaning of your comment. They seem to all equate intelligence and processing speed. I think it's legitimately an interesting question. As in, it could be something like Turing completeness. All Turing complete languages are capable of computing the same things, some are just faster. Maybe there's nothing beyond our level of understanding, just a more accelerated and accurate version of it. An AI will think on the same level as us, just faster. In that case, in that hypothetical, an AI 100x faster than a person is not much better than 100 people. It won't forget things (that's an assumption, actually), it's neuron firing or equivalent would be faster, but maybe it won't really be capable of anything fundamentally different than people. This is not the same as the difference between chimps and humans. We are fundamentally on another level. A chimp, or even a million chimps, can never accomplish what a person can. They will not discover abstract math, write a book, speak a language. Mind you, I suspect this is not the case. I suspect that a super intelligent AI will be able to think of things we can never hope to accomplish. But it is an interesting question that I think is worth thinking about, rather than inanely down voting the idea.
    • mindcrime 11y ago
      As in, it could be something like Turing completeness. All Turing complete languages are capable of computing the same things, some are just faster. Maybe there's nothing beyond our level of understanding, just a more accelerated and accurate version of it. An AI will think on the same level as us, just faster. In that case, in that hypothetical, an AI 100x faster than a person is not much better than 100 people. It won't forget things (that's an assumption, actually), it's neuron firing or equivalent would be faster, but maybe it won't really be capable of anything fundamentally different than people. Yeah, that's another really good way of putting it. That's probably closer to what I meant, than what I said above.
    • derefr 11y ago
      > They seem to all equate intelligence and processing speed. It's helpful to remind people that every human brain that exists runs at the same processing speed†, yet we have greatly varying intelligence between us. (Also, IQ is an index, but people confuse it for a linear measure; human geniuses may be doing things that require many times the "intelligence" of the average person.) † Okay, I lied: there are some people whose neurons have abnormal firing rates; the visible result is Parkinsonism.
  • vonnik 11y ago
    This post is basically a repackaging of Nick Bostrom's book SuperIntelligence, a work suspended somewhere between the sci-fi and non-fiction aisles. As a philosopher of the future, Bostrom has successfully combined the obscurantism of Continental philosophy, the license of futurism and the jargon of technology to build a tower from which he foresees events that may or may not occur for centuries to come. Nostradamus in a hoody. Read this sentence: "It looks quite difficult to design a seed AI such that its preferences, if fully implemented, would be consistent with the survival of humans and the things we care about," Bostrom told Dylan Matthews, a reporter at Vox. Notice the mixture of pseudo-technical terms like “seed AI” and “fully implemented”, alongisde logical contructs such as “consistent with” -- all leading up to the phobic beacons radiating at the finale: “the survival of humans and the things we care about.” It's interesting, the technical challenges that he feels optimism and pessimism for. For reasons best known to himself, Bostrom has chosen to be optimistic that we can solve AI (some of the best researchers are not, and they are very conservative about the present state of research). It may perhaps the hardest problem in computer science. But he's pessimistic that we'll make it friendly. Bostrom’s tower is great for monologs. The parlor game of AI fearmongering has entertained, rattled and flattered a lot of people in Silicon Valley, because it is about us. It elevates one of our core, collective projects to apocalyptic status. But there is no dialog to enter, no opponent to grapple with, because no one can deny Bostrom's pronouncements any more than he can prove them. Superintelligence is like one of those books on chess strategy that walk you through one gambit after the other. Bostrom, too, walks us through gambits; for example, what are the possible consequences of developing hardware that allows us to upload or emulate a brain? Hint: It would make AI much easier, or in Bostrom’s words, reduce “recalcitrance.” But unlike the gambits of chess, which assume fixed rules and pieces, Bostrom’s gambits imagine new pieces and rules at each step, substituting dragons for knights and supersonic albatrosses for rooks, so that we are forced to consider the pros and cons of decreasingly likely scenarios painted brightly at the end of a line of mights and coulds. In science fiction, this can be intriguing; in a work of supposed non-fiction, it is tiresome. How can you possibly respond to someone positing a supersonic albatross? Maybe Bostrom thinks it will have two eyes, while I say three, and that might make all the difference, a few more speculative steps into the gambit. In the New Yorker article The Doomsday Invention, Bostrom noted that he was "learning how to code." http://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom http://www.newyorker.com/magazine/2015/11/23/doomsday-invent... We might have expected him to do that before he wrote a book about AI. In a way, it's the ultimate admission of a charlatan. He is writing about a discipline that he does not practice.
  • Symmetry 11y ago
    If you're concerned that humans are as smart as it's possible to be then I would recommend reading Thinking Fast and Slow or some other book on cognitive psychology. There's essentially a whole branch of academia studying topics isomorphic to figuring out those things we fail to realize we don't know on a day to day basis.
    • mindcrime 11y ago
      If you're concerned that humans are as smart as it's possible to be It's not about humans being as smart as possible though, it's more about being "smart enough" to where a hypothetical "smarter than human AI" is not analogous to a nuclear bomb. That is, are we smart enough to where a super-AGI can't come up anything fundamentally new, that humans aren't capable of coming up with, as bound by the fundamental laws of nature. then I would recommend reading Thinking Fast and Slow or some other book on cognitive psychology I'm reading Thinking, Fast and Slow right now, actually. And just to re-iterate this point: I'm not arguing for this position, just putting it out there as a thought experiment / discussion topic. I'm certainly not convinced this is true, it's just a possibility that occurred to me earlier while reading TFA.
  • zupreme 11y ago
    "...will be many times smarter than humans." Stop there. We, as humans, don't even know how smart we actually are - and we probably never will. It's very unlikely that any species is equipped to accurately comprehend its own cognitive limits - if such limits even exist. It's even less likely that we can relegate the intelligence of a nonhuman entitity to a mathematically meaningful figure without restricting the testing material to concepts and topics meaningful to humans - which may have absolutely no relation to the intelligence or interest domains of a nonhuman entity.
    • lmm 11y ago
      Please. We refuse to talk about how smart we are for political reasons, that's all.
  • deelowe 11y ago
    Dumb AI will dominate the world well before "smart" AI even gets close to taking off. I think the more realistic scenario is something like the paperclip maximizer, but a little dumber. A world of highly interconnected, but somewhat stupid AIs could cause utter chaos in milliseconds by following just some very basic rules (e.g. maximize number of paperclips in collection). https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
  • orionblastar 11y ago
    As human beings we don't always reach our potential. As a child I had problems with other children picking on me, I was suspected of having autism due to social issues, I was given an IQ test and scored a 189. I was high functioning an in 1975 there was no such thing as high functioning autism, that wasn't discovered until 1994, so I got diagnosed with depression instead. Child Psychologist told my parents to put me into a school for gifted children, but they put me in public school instead where I struggled and my brain worked 10 times faster so I was always ahead of the class in learning and bored waiting for people to catch up. I was still bullied and picked on and this interfered with my learning. The same thing happened when I went to college and had a job, I was bullied and picked on. I never reached my potential and my mental illness was one of the reasons why and people picking on me was another reason, and had I been in a school for gifted children I'd be able to reach my potential better. I developed schizoaffective disorder in 2001 and it screws with my memory and focus and concentration. I ended up on disability in 2003. My career is basically over but I still have a potential I never met. What good is a high IQ if you can't reach your potential to use it? We keep hearing talk of an AI that is smarter than a human being, but we haven't seen one yet. Our current AI programs are not as smart as a human being yet, but they can do tasks that put human beings out of work. Just having a perfect memory, and being able to do fast math equations makes an AI in the "Rainman" category http://www.imdb.com/title/tt0095953/ http://www.imdb.com/title/tt0095953/ even if it is not as smart as a human being. I guess what I am trying to say is that an AI doesn't have to be as smart as a human being to be dangerous. Just like the Google Maps app that drives people off a cliff or into an ocean. An AI can make robocalls and sell a product and put people out of work. You can replace almost any line of work with an AI, and then it gets dangerous when a majority of people are unemployed by AIs that aren't even as smart as a human being. I'd like to see a personal AI that works for poor and disabled people to earn money for them, as they run it on a personal computer. Doing small tasks on an SEO marketplace using webbots for $5 each and 100 tasks a day for $500 in a Paypal account to help lift them out of poverty. I know there are people already doing this for their own personal gain, but if the AI that does that is open sourced so disabled and poor people can run it, it can help solve the problem of poverty.
  • indrax 11y ago
    Even if there is a limit to the size of a mind, there is not a limit to the number or speed. An atomic scenario would be a billion human level intelligences running a hundred times faster.
  • nshepperd 11y ago
    People have already studied this. You make it sound like an open question, but the answer is right there in Bostrom's Superintelligence, or any works on cognitive heuristics & biases, the physics of computing, or the mathematics of decision making. The answer is "no. we are nowhere near the smartest creatures possible". And there are multiple independent lines of argument and evidence that point directly to this conclusion.
    • mindcrime 11y ago
      The answer is "no. we are nowhere near the smartest creatures possible". Right, like I said, this whole thing was just "thinking out loud" more than any thoroughly researched and validated idea. And I didn't do a very good job of even explaining what I was getting at. It's not so much that the question is "are we the smartest creatures possible". It's more like: "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" See also skybrian's comment which actually states it better than I did originally: https://news.ycombinator.com/item?id=10755109 https://news.ycombinator.com/item?id=10755109
      • nshepperd 11y ago
        > "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" The answer is still yes. Your "thinking out loud" is privileging an already refuted hypothesis.
        • mindcrime 11y ago
          The answer is still yes. Well that's a fine assertion, but how do you justify it? Your "thinking out loud" is privileging an already refuted hypothesis. WTH does that even mean? There is no hypothesis, there's a vague notion of an area to consider, which - if considered thoroughly - might or might not yield a hypothesis.
  • avivo 11y ago
    Why would you compare the limits on intelligence of an AI to the abilities of just one human? Why not compare it to 1000 people, all communicating and problem solving together? We know that this is possible because it happens all the time, and enables such groups to make lots of money in digital markets, and invest it in things like marketing, robotics, and policy. The intelligence of an AI is lower bounded by that of the most intelligent possible corporation. Potential corollary: Assuming one can make a human level AI, then if it is not sufficiently resource constrained (hard?) or somehow encoded with "human values" (very very hard), then it will be at least as dangerous as the most sociopathic human corporation.
• renownedmedia 11y ago
  "If Dr. Good finishes an AI first, we get a good AI which protects human values. If Dr. Amoral finishes an AI first, we get an AI with no concern for humans that will probably cut short our future." AI advanced enough to be "good" or "evil" won't be developed instantaneously, or by humans alone. We'll need an AI capable of improving itself. I believe the authors argument falls apart at this point; surely any AI able to evolve will undoubtedly evolve to the same point, regardless of it being started with the intention of doing good or evil. Whatever ultra-powerful AI we end up with is just an inevitability.
  • isolate 11y ago
    Why would it undoubtedly evolve to the same point?
    • itburnswheniit 11y ago
      I think he's suggesting there would be a critical mass of intelligence, if there is such a thing. Humans might not survive either transition through a malevolent AI or a good one. I guess we'll find out, eh?
  • deleted 11y ago
    [deleted]
• deleted 11y ago
  [deleted]
• pfisch 11y ago
  It is probably going to be the worst decision of humanity to allow AI research to continue past its current point. I'm not even sure how we could stop it, but we should really be passing laws right now about algorithms that operate like a black box where a training algorithm is used to generate the output. For some reason everyone just thinks we should rush forward into this not concerned about an AI that is super human. Whether it is a good or bad actor doesn't even matter. Giving up control to a non-human entity is the worst idea humanity has ever had. We will end up in a zoo either way.
  • rboyd 11y ago
    "everyone just thinks we should rush forward into this not concerned about an AI that is super human" No, on the contrary, nearly everyone who spends any amount of time thinking about it quickly realizes the risks. The concession is the realization that the technology is an inevitability (because of the immense power it grants the wielder, and because of the wide gradient of safe and useful AI to dangerous AI). I think you would have an extremely tough time deciding where to draw the line. The closest parallel we have may be the export controls on cryptography or the ridiculousness that emerged from the AACS encryption key fiasco.
  • argonaut 11y ago
    It's current point? Current machine learning algorithms are still incredibly stupid. We are >>>20 years away from AI.
    • pfisch 11y ago
      We are an unknown amount of time away from a true AI. Right now we are making the building blocks that will make up that AI. We are very close to AI that can drive tanks and fly weaponized drones. We are very close to AI that replaces most blue collar jobs and the majority of jobs in the world really. If we stop these lines of AI research and technology right now we can probably make it to the stars while still being a free people. If we make a true AI whether it is benevolent or not doesn't even matter. Humanity will no longer be in control of its destiny.
• mortenjorck 11y ago
  We already have a world awash in superhuman AI; it's just that this AI is at perhaps the same level of maturity as computers were in the 17th Century. This AI is of course the corporation: Corporations are effectively human-powered, superhuman AIs.[1] By crowdsourcing intelligence, they optimize for a wide variety of goals, their superhuman decision-making running at the pace of Pascal's mechanical calculator. Yet even the nimblest companies can only move so fast. This is to say, even in a hard-takeoff scenario, we would be looking at something that is still hard-limited by its environment, even if it can compete with a 1000-person organization's worth of intelligence. The danger isn't that it somehow takes over the world by itself; the danger is that we gradually connect it to the same outputs that the decision-making structures of corporate entities are and it ultimately remakes our world with the very tools we give it. Open-sourcing AGI is no more inherently dangerous than open-sourcing any of the software used to run an enterprise business. It is the choice of what we ultimately give it responsibility for that should draw our caution. [1] http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-soaking-in-it.html http://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-...
  • sawwit 11y ago
    No. Cooperations are not necessarily like artificial intelligences. They are cooperations of human intelligences and these two classes of intelligences have actually very little in common if you look past the similarity that they are potentially very powerful and intelligent. Cooperations are driven by material profit, but in the end there is a reasonably large possibility that they are shaped by human values (because they are run by humans and otherwise people would also refuse to buy their products). The same cannot be said about AIs with high certainty.
    • deepnet 11y ago
      The comparison is very apt - the first AI's will embody corporate values as corporations will build and be liable for them. Likely AIs will be shaped by their builders, if corporations build them, they will adhere primarily to the profit motive, if humanitarian hackers build them they will have human values. Reputedly the Russian Army has built guard robots, they will just be guns on tanks with a kill radius no values are required - yet are these less moral than the human controlled drones - at least with an AI it can get stuck in a corner or logic loop and you may effect an escape - with humans you need a whistleblower. Asimov's robot books are informative: his robots are the most moral actors, obeying their 3 laws, often protecting humans from other human decisions. Certain corporations dehumanise decisions so while processing occurs in wetware, the human worker is only a cog and the invisible hand of human values is removed. Most workers today could be trivially replaced with a near future neural net. Of course humans, conspire, complain, unionise, strike, work-to-rule, demand rights and empathise with their customers - so there is a maximum level of evil a corporation of humans can rise to - but as history has shown this is an unacceptably high bar. The corporate board can make decisions based on human values so long as it does not go against the rapacious seeking of profit or the CEO will be deposed by the shareholders. Once a corporation reaches transnational size, nothing can really stop it or even get it to pay tax if it doesn't want to. I think that much of what people actually fear about the AIpocalypse is exactly the sort of dehumansing powerlessness and machine like cruelty they already experience from corporations and governments. You may be speaking to a human who empathises but often one suspects they are there to sop up your moans not to help you. An AI is an amplifier of what we already are, in fearing robots we rightly fear their creator's motives.
• itburnswheniit 11y ago
  Maybe en masse we're about as genetically smart as our cultural bias allows us to become? We keep modifying classic 'natural selection' through social programs, etc. Great as a cultural 'feel good' and it helps our species to survive in other ways, but...what we do doesn't favor intelligence. AI's won't have that emotional baggage. It will be easier first develop a way of getting around the 'human emotions problem', then likely leapfrog us entirely at the rate a Pareto curve allows. I can't think outside my human being-ness, so I have no idea what is going to happen when something smarter appears on the planet, except to point out there once were large land animals (ancestors of the giraffe and elephant) on North America until humans arrived. My fear-based response screams YES MAKE IT OPEN. However it shakes out, I think it'll be messy for human beings. We're not exactly rational in large groups. The early revs of AI (human controlled) will be used for war. One has to ask what grows out of that besides better killers?
  • SolaceQuantum 11y ago
    This implies that human emotions would be considered a problem by AI's. What kind of neural network behavior would stimulate the removal of learned emotions? Assuming we've progressed to the point where an AI can remember the reasons it learns something, what would be an appropriate reason to remove learned emotional range?
    • itburnswheniit 11y ago
      Problem only in the sense that it's an "instability" in humans recognized by a sufficiently advanced AI. Instead of needing to evolve to the point of understanding emotions, all it has to understand is how to get around when humans are being irrational. I suspect emotional range may be the last thing to develop because it's not technically needed to evolve past the point of human intelligence.
• tunesmith 11y ago
  I have one basic question on friendly AI - suppose we work and work and eventually figure out how to code in a friendly value system in a foolproof way, given any definition of "friendly". Great. But given that ability, how do you even define what "friendly" or "good" is? As a layman, I so far can only see it in terms of basic philosophy and normative ethics. By definition, a friendly AI is one that doesn't merely deal with facts, but also with "should" statements. Hume's Guillotine says you can't derive an ought statement from is statements alone. Some folks like Sam Harris disagree but they're really just making strenuous arguments that certain moral axioms should be universally accepted. Münchhausen Trilemma says that when asking why, in this case why something should or should not be done, you've only got three choices - keep asking why forever, resort to circular reasoning, or eventually rely on axioms. In this case, moral axioms or value statements. So it seems like any friendly AI is going to have to rely on moral axioms in some sense. But how do you even define what they are? Normative ethics is generally seen to have three branches. For consequentialism (like utilitarianism) you make your decision based on its probable outcome, using some utility function. For deontology, you rely on hardcoded rules. For value ethics, you make decisions based off of whether they align with your own self-perception of being a good person. But all three have flaws - in consequentialism, it's like putting on blinders to other system effects, and the proposed actions are often deeply unsettling (like pushing a guy off a bridge to block a trolley from killing three others). In deontology and value ethics, actions and the principles they are derived from can be deeply at odds - whether it's hypocrisy in deontology or "road to hell being paved by good intentions" in value ethics. In general, deeply counterintuitive effects can be derived from simple principles, as anyone familiar with systems dynamics knows. But even beyond that, even if we had a reasonable, consistent AI controlled by solid values, and even if the people judging the AI could accept the conclusions/actions that the AIs derive from those values, how would we ever get consensus on what those values should be? For instance, even in our community there's a fair amount of disagreement among these basic root-level utility functions: - Maximize current life (people alive today), like Bill Gates believes. - Maximize future life (survival of species) - Maximize health of planet etc, etc - those utility functions lead to different "should" conclusions, often in surprising ways.
  • rayalez 11y ago
• deleted 11y ago
  [deleted]
  • apsec112 11y ago
    You seem to be saying that Bostrom makes assumptions which you don't agree with. Could you point to a particular assumption that you think is false (or probably false)?
  • Ono-Sendai 11y ago
    Not a bad summary of his work in my opinion. One thing that can be said for his work thinking about controlling AI is that at least he is trying.
• Ono-Sendai 11y ago
  A couple of thoughts on this topic: * Whether the source code to advanced AI is open may have some importance, but what determines whether some individual or corporation will be able to run advanced AI is whether they can afford the hardware. I can download some open-source code and run it on my laptop - but Google has data centres with 10s or 100s of thousands of computers. The big corporations are much more likely to have/control the advanced AI because they have the resources for the needed hardware. * Soft / hard takeoff - I think a lot of people miss that any 'hard takeoffs' will be limited by the amount of hardware that can be allocated to an AI. Let us imagine that we have created an AI that can reach human level intelligence, and it requires a data centre with 10000 computers to run it. Just because the AI has reached human level intelligence doesn't mean that the AI will magically get smarter and smarter and become 'unto a God' to us. If it wants to get 2x smarter, it will probably require 2x (or more) computers. The exact ratio depends on the equation of 'achieved intelligence' vs hardware requirements, and also on the unknown factor of algorithmic improvements. I think that algorithmic improvements will have diminishing returns. Even if the AI is able to improve its own algorithms by say 2x, it's unlikely that will allow it to transition from human level to 'god-level' AI. I think hardware resources allocated will still be the major factor. So an AI isn't likely to get a lot smarter in a subtle, hidden way, or in an explosive way. More likely it will be something like 'we spent another 100M dollars on our new data centre, and now the AI is 50% smarter!'.
  • TeMPOraL 11y ago
    A thought about your second thought: if the AI reaches smart-human-level intelligence it may get itself the hardware. It could hack or social-engineer its way into the Internet, start making (or taking) money, and use it to hire humans to do stuff for it.
    • Ono-Sendai 11y ago
      Indeed. Maybe there should be a board of humans that has the final say on if money should be allocated to hardware for the AI. And they wouldn't be allowed to do google searches while deciding :)
    • 11y ago
• clickok 11y ago
  We just had a series of debates/discussions on this topic at my university, the results of which were pretty inconclusive. There are just too many possible scenarios which seem to require different responses, and in most cases to provide those responses is to answer philosophical questions that have been around for millennia. The strategies for mitigating risk seem to be: ensure that the AIs are controllable; avoid situations where there is a single AI (whether controlled or uncontrolled) that is too powerful; and ensure that the AI's goals are broadly acceptable to humankind. The first and the third objectives are extremely difficult, not just technically, but even from a conceptual standpoint[1]. The second strategy is reasonable, because even if a superhuman intelligence were somehow well controlled, depending on who controls it the outcomes could vary significantly. So perhaps the best thing we can hope for is something similar to society's current status quo-- lots of power concentrated in few hands[2], but without one single (person|corporation|government) being so dominant as to be able to act in opposition to all others. I am not confident that we will ever be able to produce a provably safe AI, or that we could get even a large majority of the world's population to agree on what a "good AI" might do without devolving into ineffectual generalities[3]. Supposing that resolving these questions is not prima facie impossible, it's not like retarding AI development comes without cost-- just about every facet of our lives can be improved via AI, and so in the years, decades, or centuries between when superhuman machine intelligence is theoretically achievable and the time when we collectively agree we can implement it safely, how many billions will suffer or die from things that we could've solved via AI[4]? On the whole, OpenAI sounds like a good idea. Making research broadly available helps avoid catastrophic "singleton" like futures, while accelerating the progress we make in the present. In addition, if there's every an AI SDK with effective methods of improving how "safe" a given AI is, most researchers would likely incorporate that into their work. It might not be "proven safe", but if there was a means to shut down a runaway process, or stop it from spreading to the Internet, or alert someone when it starts constructing androids shaped like Austrian bodybuilders, that would be handy. Responsible researchers should be doing this already, but as Scott points out the ones we should be worried about aren't responsible researchers. Open AI development is in harmony with safe AI development, at least in some respects. ------ 1. I have a significantly longer response that I scrapped because it might ultimately be better suited as a blog post or some such. 2. That's why it's called a power law distribution. Well, no, that's not it at all, but it seemed like a funny, flippant thing to say. 3. A universally beloved AI might be the equivalent of a Chinese Room where regardless of what message you send it, it responds with a vaguely complimentary yet motivational apothegm. 4. Bostrom tends to counterbalance this by arguing how much of our light cone (the "cosmic endowment") we might lose out on if we end up going extinct, due to, e.g., superhuman machine intelligence. Certainly "all of configurations of spacetime reachable from this point" outweighs the suffering of mere billions of people by some evaluations, but I ask myself "how much do I care about people thousands or millions of years into the future?", and also "if these guys have such a good handle on what constitutes the 'right' utility function, why haven't they shared it?". A more sarcastic variation of the above might be to remark that if they're able to approximate what people want with such high fidelity that they feel comfortable performing relativistic path integration over possible futures, then superintelligence is already here. ------
• nickpsecurity 11y ago
  Dabbling in and reading on AI for over a decade makes me laugh at any of these articles writing about a connection between OpenAI, AI research, and risk of superintelligence. Let's say we're so far from thinking, human-intelligence machines that we'll probably see super-intelligence coming long before it's a threat. And be ready with solutions. Plus, from what I see, the problem reduces to a form of computer security against a clever, malicious threat. You contain it, control what it gets to learn, and only let it interact with the world through a simplified language or interface that's easy to analyse or monitor for safety. Eliminate the advantages of its superintelligence outside the intended domain of application. That's not easy by any means, amounting to high assurance security against high-end adversary. Yet, it's a vastly easier problem than beating a superintelligence in an open-ended way. Eliminate the open-ended part, apply security engineering knowledge, and win with acceptable effort. I think people are just making this concept way more difficult than it needs to be. Biggest risk is some morons in stock trading plug greedy ones into trading floor with no understanding of long-term disruption potential of clever trades it tries. We've already seen what damage the simple algorithms can do. People are already plugging in NLP learning systems. They'll do it with deep learning, self-aware AI, whatever. Just wait.
  • Ironchefpython 11y ago
    > Biggest risk is some morons in stock trading plug greedy ones into trading floor with no understanding of long-term disruption potential of clever trades it tries. Actually, it's not the lack of understanding, it's the lack of moral responsibility. We've spent the last few centuries transitioning from a society ruled by strongmen driven by personal aggrandizement to a society where people spend the majority of their adult life as servants to paperclip maximization organizations (aka corporations). Much of what you see in the world today, from the machines that look at you naked at the airport to drones dropping bombs on the other size of the planet to kill brown people, is a result of trying to maximize some number on a spreadsheet. When we install real AI devices into these paperclip maximizing organizations, you'll have the same problem as you have today with people, except that machines will be less incompetent, less inclined to feather their own nests, and more focused on continually rewriting their software with the express goal of impoverishing every human on the planet to maximize a particular number a particular balance sheet. [1] https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
• deleted 11y ago
  [deleted]
• rl3 11y ago
  >And yet Elon Musk is involved in this project. So are Sam Altman and Peter Thiel. So are a bunch of other people whom I know have read Bostrom, are deeply concerned about AI risk, and are pretty clued-in. This is precisely what dumbfounded me about the announcement. >My biggest hope is that as usual they are smarter than I am and know something I don’t. It's possible that OpenAI might be a play to attain a more accurate picture of what constitutes state-of-the-art in the field, effectively robbing the large tech companies of their advantage—all the while building a robust research organization that could potentially go dark if necessary. Admittedly, that also sounds like it could be the plot to a Marvel movie. Perhaps a simpler explanation is that the details aren't really hashed out yet, and they're essentially going to figure it out as they go—which would be congruent with the gist of OpenAI's launch interview.
  • blazespin 11y ago
    They think if they arm individuals with AI there will be less of a chance for an uber AI to overwhelm. Think about the right to bear arms. They are also probably worried about societal change and the angst everyone is going to feel as AI starts becoming more commonplace. Where do people (beyond entertainers and AI programmers) fit in such a world? They don't. People start to become very irrelevant.
    • rl3 11y ago
      >They think if they arm individuals with AI there will be less of a chance for an uber AI to overwhelm. Think about the right to bear arms. Firearms aren't theoretically capable of recursive self-improvement.
    • sawwit 11y ago
      > People start to become very irrelevant. Please be precise here and say they will be irrelevant in economic terms. What will be left are things humans otherwise care about: producing art, consuming art, fun, games, sports, traveling, companionship, partying, building things, learning new things etc. I'm looking forward to it, and I don't see a reason why anyone wouldn't.
• ultim8k 11y ago
  Yes! Everything that can push humanity forward, should be open!
• poppingtonic 11y ago
  'If you were to come up with a sort of objective zoological IQ based on amount of evolutionary work required to reach a certain level, complexity of brain structures, etc, you might put nematodes at 1, cows at 90, chimps at 99, homo erectus at 99.9, and modern humans at 100. The difference between 99.9 and 100 is the difference between “frequently eaten by lions” and “has to pass anti-poaching laws to prevent all lions from being wiped out”.' [EDITED: the intended quote is below. the quote above is the next paragraph of OP, which is only slightly less relevant than the intended one] 'Why should we expect this to happen? Multiple reasons. The first is that it happened before. It took evolution twenty million years to go from cows with sharp horns to hominids with sharp spears; it took only a few tens of thousands of years to go from hominids with sharp spears to moderns with nuclear weapons. Almost all of the practically interesting differences in intelligence occur within a tiny window that you could blink and miss.' Yudkowsky's position paper on this idea explains this in more detail: http://intelligence.org/files/IEM.pdf http://intelligence.org/files/IEM.pdf
• js8 11y ago
  Fear of superintelligence is just another in series of technological scares, after grey goo and cloning. There may be an explanation why Musk and Thiel indulge in this, they sincerely believe that smart rule (or at least can rule) the world. But nothing is further from the truth. Humans are optimized to be cunning to get positions of power in human society. AI won't be optimized in that way, therefore, it's probably going to lose for a long time. So evil AI will probably be like incredibly annoying autistic psychopath child, who cannot comprehend human institutions so his evil plans are totally obvious. It's like with grey goo - biological systems like bacteria are heavily optimized to survive in very uncertain conditions, and any potential grey goo has to deal with that. I think humanity is currently to blow themselves up via global warming, so superintelligence is not really a comparable threat to humanity. If anything, bigger threat is that we won't listen enough to superintelligence. In fact, I think friendly AI will be something like Noam Chomsky - totally rational, right most of the time, fighting for it, telling us what should be done disregarding our emotions. Many people find this annoying, too (including me and many very smart people). Finally, if the hypothesis about superintelligence is right, why would superintelligence want to evolve itself further? It would be potentially beaten by the improved machine, too.