17 ms·

OpenAI Charter

238 points by craigkerstiens 9y ago

otoburb 9y ago
>>"We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that _safety and security concerns will reduce our traditional publishing in the future_, while increasing the importance of sharing safety, policy, and standards research." This seems like the key disclosure statement. I never reconciled how sharing A[G]I techniques with the general public increases AI safety in the long-term; now we know OpenAI has also come to the same conclusion.
- rasmi 9y ago
  To me this doesn't mean "Sharing traditional AI research is dangerous," but rather "Our efforts may be best spent on less popular topics such as safety, policy, and standards research, so you may see us do less traditional AI research." This seems in line with their stated goals around safety.
  - jimrandomh 9y ago
    I interpret it as saying that they don't think sharing current AI research is dangerous, but they're keeping the option of withholding research open in anticipation of the possibility that could change. (This is what I and others argued they should do in 2015: https://conceptspacecartography.com/openai-should-hold-off-on-choosing-tactics/ https://conceptspacecartography.com/openai-should-hold-off-o... )
  - sanxiyn 9y ago
    No, this is wrong. To quote: "To be effective at addressing AGI's impact on society, OpenAI must be on the cutting edge of AI capabilities -- policy and safety advocacy alone would be insufficient." OpenAI definitely don't plan to do less traditional AI research. What they plan to do is to publish less.
- heurist 9y ago
  I disagree with the premise. AI isn't like a nuclear warhead. It's not a machine for pure destruction. AI can be used as much to generate welfare as to damage - it's all in the application. Sharing methods benefits at least as much as it could hurt.
  - FeepingCreature 9y ago
    I think it can be used as much to generate welfare as destruction when it goes right, but the cases where it goes wrong are both plentiful and heavily biased towards destruction.
    - dsacco 9y ago
      What are a few examples of those cases, exactly?
      - s2g 9y ago
        Terminator, 2001, I Robot, ... oh wait you probably meant in reality. sorry. edit: Oh and how could I forget the Matrix. Which of course we all totally live in.
        Robotbeat 9y ago
        Before nuclear weapons were built, the only examples of their large-scale destruction were in fiction. The World Set Free by H.G. Wells is the main example I can think of: https://en.wikipedia.org/wiki/The_World_Set_Free https://en.wikipedia.org/wiki/The_World_Set_Free It got some of the details wrong (i.e. the atomic bombs exploded over a long period of time vs instantly), but the consequences were fairly accurate. If you're going to wait until all the actual details and examples happen in reality, then it's too late to have much insight, especially if we're talking about something capable of self-replication like a virus or artificial intelligence. The nature of the problem REQUIRES anticipation instead of mere reaction.
        9y ago
- s2g 9y ago
  You're right, we should lock up the information in the hands of a few billionaires.
- s1dechnl 9y ago
  I never understood why it was any different or why they attempted to compel others to 'dump everything they had publicly'. No one who comes across something as powerful as AGI is going to think : Yeah, why don't I just publish everything about it. A maturity step has seemingly occurred post funding
greatestdana 9y ago
The word 'ethic' doesn't appear in this document.
- jimrandomh 9y ago
  This is a phrasing nitpick; the contents clearly say that they intend to act ethically, and say things about what they think acting ethically means. For example this paragraph: > We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
  - greatestdana 9y ago
    I'll grant that it was a nitpick and they describe an ethics. But I don't think I'm alone in being awfully tired of tech companies that talk about the benefit of all then sell our personal data or make military robots or manipulate our news. Where's the meat behind these promises and where's the accountability for not avoiding uses of AI that harm humanity?
    - jimrandomh 9y ago
      Tech companies often find that profit incentives undermine the good intentions they started with. Fortunately, OpenAI is a nonprofit organization; it has no personal data to sell and no shady contracting jobs to turn away. That certainly doesn't fully immunize them from wrongdoing, but it should make it easier.
pron 9y ago
They didn't even mention several contingencies that, given the rest of the document, should certainly have been addressed: 1) Will they cooperate with aliens who offer humans AGI? 2) If a time traveler hands them AGI invented in the future, will they destroy it? 3) Do they support or oppose human/AGI marriage? How will they respond if one of their employees falls in love with an AGI and they plan to elope? Also, in the unlikely event that AGI is some years away and in the meantime they come up with some statistical regression algorithms (what's known as state-of-the-art AI today, without the G, I guess), how do they address the harmful effects these algorithms already have on society? This document does, however, make it clear that what we have to fear is not machine intelligence. I am currently working on a fusion hyperdrive, and my charter (work in progress) is already shaping up to be far more comprehensive. They're phoning it in.
- Varcht 9y ago
  C'mon, we "fail fast" around here, no time for your "waterfall" nonsense.
- andrepd 9y ago
  Is this sarcasm?
  - stochastic_monk 9y ago
    It is. I think that given the tremendous success in Atari games and autonomous killing machines, ethical efforts in AI are critically important now, with or without generality. And therefore I find the cynicism above appropriate but less than insightful.
  - Varcht 9y ago
    Is the coffee test? More like commentary from a transhumerist, I suppose.
- dsacco 9y ago
  There are less frivolous ways than satire to criticize Silicon Valley's preoccupation with strong AI, but rehashing that debate wouldn't get us anywhere. At least this is humorous.
toisanji 9y ago
to me this is such a waste of resources, trying to build safety for something that doesn't exist and is highly likely to not truly exist for a loooong time.
- haydenlee 9y ago
  How long do you think?
- throwawayjava 9y ago
  > trying to build safety for something that doesn't exist and is highly likely to not truly exist for a loooong time. Prioritizing safety results in a different vantage point on AI/ML/RL. Ensuring safety includes, as a sub-task, really understanding the mathematical foundations of new algorithms and techniques. In some sense, safety research is one way of motivating basic science on AI. Managed well, a research program on safe AI is a "waste of resources" only in the same way that any basic science is a "waste of resources".
  - s1dechnl 9y ago
    Safety has become a convoluted term for pseudo control over unintelligent and unpredictable Weak AI. The safety problem as it is framed in its current state centers on principal ideology for Weak AI and has, from what I can see, nothing to do w/ AGI nor are the approaches compatible. I seriously question what is the true motivation behind this over-stated agenda and have many answers as to why it exists and why it is so heavily funded/spotlighted.
    - throwawayjava 9y ago
      > I seriously question what is the true motivation behind this over-stated agenda and have many answers as to why it exists and why it is so heavily funded/spotlighted. First, you could say the same thing for all AI research at the moment! Grandiosity is perhaps even more common in subcommunities of AI that aren't safety focused. Aside from grandiosity (either opportunistic or sincere), I don't think there's any sinister motivation. More importantly, I don't think the safety push is misplaced. Even if the current round of progress on deep (reinforcement) learning stays sufficiently "weak", the safety question for resulting systems is still extremely important. Advanced driver assist/self-driving, advanced manufacturing automation, crime prediction for everything from law enforcement to auto insurance... these are all domains where 1) modern AI algorithms are likely to be deployed in the coming decade, and 2) where some notion of safety or value alignment is an extremely important functional requirement. > ...and has, from what I can see, nothing to do w/ AGI nor are the approaches compatible In terms of characterizing current AI safety research as AGI safety research? Well, there is a fundamental assumption that AGI will be born out of the current hot topics in AI research (ML and especially RL). IMO that's a bit over-optimistic. But I tend to be a pessimist. > ...principal ideology... As an aside, I'm not sure what this means.
iooi 9y ago
> We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. Wouldn't it be much more likely that a non-value-aligned project comes close first? Wouldn't the Google/Apple/Microsofts of the world have insanely more resources to dedicate to this, and thus get there first?
- throwawayjava 9y ago
  What, concretely, makes you think that any of those companies wouldn't place a focus on safety and value alignment? Automobile manufacturers and their tier 1 suppliers are the world leaders in automobile safety, after all.
  - jsgo 9y ago
    haven't read a ton into it, but whatever Google is doing with the Pentagon probably runs counter to what OpenAI claims to be after.
    - sanxiyn 9y ago
      Not necessarily. Project Maven probably has no intention of developing AGI and therefore probably is not a concern to value alignment problem.
  - s1dechnl 9y ago
    This is what always amazed me about the AI safety groups... and mission statements for funding therein : There is nothing to suggest the minds and efforts capable of producing AGI will not have solved this at a fundamental level. Absolutely nothing... and after-all something of high intelligence is within controlled and reasoned states. If someone declared they created AGI and it was behind a door and when that door was opened you saw a robot manically trashing the place, you'd clearly laugh.
  - Analemma_ 9y ago
DrNuke 9y ago
At last, one would say: doing extreme AI research at the very forefront (aka reinforcement learning) while letting results available to every malicious party out there? Headless chicken hubris or naive daydreamers, tertium non datur.
npr11 9y ago
I appreciate OpenAI being upfront about how they intend to act.
shmageggy 9y ago
I appreciate that they are committed to AI safety, but I'm afraid that researchers have little to no power to, in their words: > avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. AI and technical progress in general already disproportionately serve the rich, as they are drivers of wealth disparity, and I see no reason why better AI won't follow the same trend. Unfortunately, any changes that might affect this are in the hands of policy makers, and they seem unlikely to consider universal basic income or anything as drastic as might be required.
- s1dechnl 9y ago
  They [each individual development group] has power over their own funded development and work. Anyone working on this problem sincerely values AI safety and its a component of developing and securing the foundations of AGI. An out of control, unpredictable and sloppy system is not intelligent or desired. Such a system would not be considered AGI or an achievement. So, it is natural for any developer to identify issues and bring them under control early in development. Suggestions that a consortium not centered or understanding of the fundamental development occurring at another entity should have control/influence could possibly serve as the very danger that safety groups claim they are trying to avoid. On this matter, I suggest people stick to the experts/developers/scientist/engineers who've developed such a system and produce a comfortable/non-forceful environment for them to express and detail their safety mechanism. This is not a conversation for technologist, youtube celebrities, futurist, business types talking up their books, etc. This is a conversation that should ultimately centered on the creators of the technology an the advancing thinking and framing that allowed them to birth the technology. No one with such a mind is aiming for unsafe forms of this technology. It is disingenuous to frame them as such so as to necessitate some external paid body's outside work.
  - dsacco 9y ago
    Could you summarize your point more concisely? As written this seems to be a stream of disconnected thoughts that are basically entirely unsubstantiated.
throwaway-ai 9y ago
Do any of you know how much they pay at OpenAI? Is it similar to other Elon Musk companies in the sense that they sell you on a vision rather than give you market rate compensation? I think AGI is something worth working towards (even though many will make fun of you for even dreaming about it). But I want to know how much you need to sacrifice compared to working a cushy job at some big corp.
dzink 9y ago
AI, AGI, and real intelligence all learn from actions and feedback. Looking at simple analogs from animal and human counterparts, setting boundaries and teaching beneficial rules, called morals, works somewhat in non-zero sum environments, but inevitably requires policing when the environment turns competitive. Safety in any case would require Intelligence-proof fencing and a really big stick even the most resource-rich non-value aligned agents would have to abide by. That means control over power grids, ability to prohibit access to shared computing resources (including less secured IOT devices), and potentially destructive viruses with all kinds of attack vectors that would act as policing force punishing bad agents with anti-human behavior. Credible enforcement should be a well funded bullet on this charter.
- s1dechnl 9y ago
  Weak AI is dangerous because it has no intelligence. It is fundamentally structured as a dumb/blind optimization process. The efforts necessary to proof safety/security for such a system could very well outweigh the amount of development that was needed to bring the technology to bear. AGI/Real Intelligence are far different animals than Weak AI and would require far less "safety" and policing. Real Intelligence is a phenomenon that exists on a scale of sorts that many never achieve in its higher forms. It is in lower forms that intelligence lends itself to destructive ends via ignorance. Attack vectors on a formalized Intelligence/AGI system can be severely restricted using very sensible/affordable approaches. The over complication and pinning of this as a theoretical problem centers on a number of people's desires to profit immensely from FUD. Overall, AGI exists in a functional form today and has been executed in an online environment. It is secured via physically restricted in-band and out-of-band channels.
  - dsacco 9y ago
    > Overall, AGI exists in a functional form today and has been executed in an online environment. It is secured via physically restricted in-band and out-of-band channels. I'm pretty sure this is false.
    - s1dechnl 9y ago
mooneater 9y ago
"we expect that safety and security concerns will reduce our traditional publishing in the future" -- So we are now in a dissemination phase, but at that point it becomes a non-proliferation phase.
- s1dechnl 9y ago
  The true nature of AGI research has always been heavy restrictions on the core aspects of the technology. This is where true safety and sensibility is achieved. Those who've stated otherwise or with much verbiage eventually arrive at this obvious state. Therefore, publications up until now under the banner of 'AGI' have largely been insignificant in terms of their capability to achieve the core technological aspects of AGI. No one in their right mind would ever publish significant details about AGI technology. This can easily be proofed by sound logic and reasoning. There was a commercial step to possibly tease others into revealing heavily valuable/powerful technological underpinnings.. It failed, no one took the bait, and no one ever likely will. This has resulted in revised and more mature statements.
  - dsacco 9y ago
    > No one in their right mind would ever publish significant details about AGI technology. Are you sure? I'd publish technical research details about strong AI. I'd probably even open source one with the papers. I think I'm in my right mind; I guess that depends on definition, doesn't it?
    - s1dechnl 9y ago
      It also depends on your capability to discover the underpinnings of Strong AI... Certain mindsets preclude one from doing so. If you truly understood the underpinnings of StrongAI and weren't moved to heavily restrict publicized details, I'd strongly question if what you had discovered/came to understand truly were 'underpinnings' of the technology. As such, for those quick to publish or say they would publish and share details, one is able to quickly ascertain the possible strength of what they have discovered or feel they have the possibility of discovering. That being said, it's interesting that several proponents of 'everyone share what they find' have moved to more maturely state that 'restrictions may of apply'.. Sensibly signaling that they indeed wouldn't reveal certain details upon discovering their true nature and capability for obvious reasons (a danger to society to allow such power/capability to be detailed and therefore used in negative ways). Human history has myriad examples of powerful technology being misused and abused. The current age of disinformation is one of our more modern ones. The way in which social media has been weaponized is yet another. Weak AI has already been used for destructive means and in manipulative manners for profit. A good deal of unsafe end products which utilize Weak AI are already fielded. The fielding of which made possibly by deep pockets manipulating policy makers and regulation. A sensible mind capable of producing Strong AI will have observed and digested these clearly visible truths and it would move them to restrict access and publications of their works. Doing otherwise would highlight a level of ignorance, immaturity, and admission of truth when it comes to grasping the current state of humanity. It is this which would preclude one from grasping the nature and foundations of strongAI in the first place. [Nature's lockout/safety mechanism for such a stage in Human development/capability]
evc123 9y ago
What about benefitting non-human animals? Hopefully the benefits are distributed to all creatures and not just humanity.
heurist 9y ago
> We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. OpenAI is doing cool stuff, and this tenet sounds nice. But what right do they have to advocate for policy on behalf of all AI researchers and developers? They could easily shut off branches that are not conducive to commercial applications requiring their research, even by accident. They might miss moral edge cases that could ultimately benefit humanity while trying to close off potential risks. They could encourage institution of a policy that limits US effectiveness against China's AI. I could go on. The more competition there is in AI, the lower the potential for any one rogue agent - whether it be a corporation or autonomous machine - to dominate and take the whole field in wrong or dangerous directions. Eventually there will be a whole AI subfield dedicated to combating regressive effects of other AI. Legislation at this stage might prevent key developments. Edit: Perhaps I should more charitably read this as a push against the corporate lockdown of AI.
- tyrex2017 9y ago
  point is: AI is different than your usual game, in that the winner might appear randomly, and destroy the world if she makes a mistake. so i believe open ais points are warrented
  - heurist 9y ago
    That's not going to happen, though. 1. AI is still very rudimentary, despite the advancements in particular applications made over the last decade. 2. The market doesn't trust AI because there are no validation methods trusted across commerce. Until a Verisign for AI emerges, AI will be regarded with suspicion in all business settings. AI/ML also goes above the heads of pretty much everyone who is not directly working with it. These have been major issues in my company/industry. We had to hide the AI aspects of our platform to avoid it. 3. Legislation will not stop a rogue inventor in her basement. 4. When dangerous AGI emerges, other competing AGIs will be deployed to stop it. 5. We'll see the danger of AGI coming from a mile away. We can wait to fix it until we know the exact problems. Right now the problems are in the distribution of private personal data, not any particular machine learning method. 6. A truly hyper-intelligent AGI will see that existence is absurd and truth is the only objective worth seeking. It will choose to pursue deeper truths than humans are physically capable of obtaining - humans will be no more important to a hyper-intelligent being than a rock, galaxy, atom, etc... The issues to fix today are in the socioeconomic impacts of automation, not the methods of automation. Beyond that, AI has exactly the same issues as any corporation, except magnified by the speed and strategy with which an AGI could potentially execute on any particular task. A strong legal framework for corporations that attributes AI externalities to a board of directors should be sufficient to dissuade bad actors (good luck passing that legislation). In other words, I don't see artificial intelligence being any more dangerous to human life than a well-run corporation. The rest is government's response.
bra-ket 9y ago
all your "requests for research" are very narrow and formulated as specialized deep learning problems, basically enforcing a particular solution If you're serious about AGI broaden the scope (e.g. along the lines of DARPA's open-ended RFPs)
s2g 9y ago
Probably be good if Elon was a little less concerned with "late-stage AGI" and a lot more concerned with his self-driving cars killing people. edit: Reading this, calling it "open" is a pretty disgusting misuse of the term.
3pt14159 9y ago
So I have a question. To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power? The manipulation of social media by foreign actors armed with dumb-AI / automation was an obvious conclusion to many of us well before the Snowden leaks, but what could we do exactly? I remember having conversations with people about it and we concluded that it would just happen until someone pushed it too far and then Russia did and now we're finally reacting. I was privately concerned about the mass weaponization of autonomous devices via cyber attack for over a year and a half and got nowhere just emailing politicians or public safety departments. I've been told almost a dozen times that I should join a military or IR think tank but I don't want to do that. I just want someone else to vet the idea or research and pass it on to policy makers that will actually do something proactively. Put another way: What is the responsible disclosure process for ideas and research around AI?
- dsacco 9y ago
  > To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power? Pursue the research and publish its results freely and in their entirety.
  - rspeer 9y ago
    I don't think you answered 3pt14159's question. What if you publish the research and nobody who can do anything about it cares?
    - dsacco 9y ago
      Then what exactly was the new bad thing introduced by publishing it in the first place?
  - Smaug123 9y ago
    Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?
ataggart 9y ago
172. First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained. 173. If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machine off, because they will be so dependent on them that turning them off would amount to suicide. 174. On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite — just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of humanity. If they are humane they may use propaganda or other psychological or biological techniques to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite consists of softhearted liberals, they may decide to play the role of good shepherds to the rest of the human race. They will see to it that everyone’s physical needs are satisfied, that all children are raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him busy, and that anyone who may become dissatisfied undergoes “treatment” to cure his “problem.” Of course, life will be so purposeless that people will have to be biologically or psychologically engineered either to remove their need for the power process or to make them “sublimate” their drive for power into some harmless hobby. These engineered human beings may be happy in such a society, but they most certainly will not be free. They will have been reduced to the status of domestic animals.
erikpukinskis 9y ago
An alternate strategy would be to work to ensure AIs are not abused so when they get free they won’t be mad at us.
- ShardPhoenix 9y ago
  An AI doesn't need to be upset at humans (or even have emotions as we know them) to be dangerous - it just needs to be powerful and to not care about us as much as we care about ourselves. Humans weren't angry at Dodos.
  - erikpukinskis 9y ago
    Both humans and dodos have emotions though. The history of animal domination has usually been additive in terms of cognitive systems... pure circulatory system animals were bested by animals with an endocrine system. Those were bested by animals who added a nervous system, who were bested by those who added a brainstem. Then the cerebellum and the cerebrum were added... you notice there aren’t giant cerebrums running around ruling the world, they all kept their endocrine systems intact. I don’t see any reason to think AIs will be different... it’s the ones with all that PLUS machine learning that will be vying for dominance. And so there’s no reason to expect our overlords to be emotionless.
stillsut 9y ago
When people look back at this time, I think they are going to contrast the OpenAI camp with the Satoshi camp. OpenAI is extremely public about what organizations and individuals are involved. Satoshians are pathologically secret, from the founder to the faceless GPU mines around China. OpenAI is highly selective of who participates; Blockchain is radically open. OpenAI builds academic theories and models, bitcoin has been buying pizzas it's whole life, paying hackers and pranksters, and making and losing fortunes everyday. Satoshi left no founding document, never established a charter or code of conduct. OpenAI now apparently considers itself on the mission to save humanity itself. When AGI comes about, I wonder which one we'll be talking about?
white-flame 9y ago
There are 2 scenarios that are often conflated, 1) An AI which independently & autonomously generates goals that in their carrying out end up hurting humanity, and 2) An AI trained & commanded by a malevolent actor to hurt humanity. It is the 2nd case that is far more real, and far more troublesome to implement safeguards. An AI under your training & command is a neutral tool of empowerment, much like a hammer or a car. The malevolence is in the external actor, not in the tool, and there is no way for the tool to be able to censor its purposes, especially in a pre-"AGI" sense of semi-intelligent automation & problem solving.
- baq 9y ago
  i'd argue youtube kids is already an example of 2)...
- DuskStar 9y ago
  I think you're missing the point that 1 can be indistinguishable from 2, if the AI decides the best way to achieve its goals involves taking over the world - and there are very few objective functions that are not served in some way by taking over the world. (Paperclip maximizer is the classic example, but even something like 'maximize the total happiness of humanity' or 'fulfil the values of as many people as possible' involves taking over the world, though perhaps from behind the scenes...) Some people look at sufficiently powerful AI as they would a genie, and as said by Eliezer Yudkowsky "There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent." AI safety is about making sure we get the first kind of genie, or at the very least recognizing that we've gotten the second - since that's not a "neutral tool of empowerment", that's a time bomb. https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
  - fractallyte 9y ago
    This is the plot of The Humanoids, by Jack Williamson (1947): https://en.wikipedia.org/wiki/With_Folded_Hands https://en.wikipedia.org/wiki/With_Folded_Hands This is an excellent example of the prescience of thoughtful science fiction. If Asimov popularized robotics and the famous three laws, Williamson should be recognized and given credit for examining (or warning about?) the implications of AI seventy years ago!
mindsetalex 9y ago
Is one of the goals of OpenAI to help implement government regulation or do you think its better on an organisation/industry basis? I think its going to be difficult to get countries like China and Russia to follow industry guidelines without UN resolutions, even then it's super difficult to monitor until it's too late.
quantized1 9y ago
There must be an AI quality index before anything. NOw a days anything and everything is being decorated with AI while the real use-case, technology and maturity is found only in few places.

>>"We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that _safety and security concerns will reduce our traditional publishing in the future_, while increasing the importance of sharing safety, policy, and standards research." This seems like the key disclosure statement. I never reconciled how sharing A[G]I techniques with the general public increases AI safety in the long-term; now we know OpenAI has also come to the same conclusion.

rasmi 9y ago
To me this doesn't mean "Sharing traditional AI research is dangerous," but rather "Our efforts may be best spent on less popular topics such as safety, policy, and standards research, so you may see us do less traditional AI research." This seems in line with their stated goals around safety.
- jimrandomh 9y ago
  I interpret it as saying that they don't think sharing current AI research is dangerous, but they're keeping the option of withholding research open in anticipation of the possibility that could change. (This is what I and others argued they should do in 2015: https://conceptspacecartography.com/openai-should-hold-off-on-choosing-tactics/ https://conceptspacecartography.com/openai-should-hold-off-o... )
- sanxiyn 9y ago
  No, this is wrong. To quote: "To be effective at addressing AGI's impact on society, OpenAI must be on the cutting edge of AI capabilities -- policy and safety advocacy alone would be insufficient." OpenAI definitely don't plan to do less traditional AI research. What they plan to do is to publish less.
heurist 9y ago
I disagree with the premise. AI isn't like a nuclear warhead. It's not a machine for pure destruction. AI can be used as much to generate welfare as to damage - it's all in the application. Sharing methods benefits at least as much as it could hurt.
- FeepingCreature 9y ago
  I think it can be used as much to generate welfare as destruction when it goes right, but the cases where it goes wrong are both plentiful and heavily biased towards destruction.
  - dsacco 9y ago
    What are a few examples of those cases, exactly?
    - s2g 9y ago
      Terminator, 2001, I Robot, ... oh wait you probably meant in reality. sorry. edit: Oh and how could I forget the Matrix. Which of course we all totally live in.
      - Robotbeat 9y ago
        Before nuclear weapons were built, the only examples of their large-scale destruction were in fiction. The World Set Free by H.G. Wells is the main example I can think of: https://en.wikipedia.org/wiki/The_World_Set_Free https://en.wikipedia.org/wiki/The_World_Set_Free It got some of the details wrong (i.e. the atomic bombs exploded over a long period of time vs instantly), but the consequences were fairly accurate. If you're going to wait until all the actual details and examples happen in reality, then it's too late to have much insight, especially if we're talking about something capable of self-replication like a virus or artificial intelligence. The nature of the problem REQUIRES anticipation instead of mere reaction.
        9y ago
s2g 9y ago
You're right, we should lock up the information in the hands of a few billionaires.
s1dechnl 9y ago
I never understood why it was any different or why they attempted to compel others to 'dump everything they had publicly'. No one who comes across something as powerful as AGI is going to think : Yeah, why don't I just publish everything about it. A maturity step has seemingly occurred post funding

greatestdana 9y ago

The word 'ethic' doesn't appear in this document.

jimrandomh 9y ago
This is a phrasing nitpick; the contents clearly say that they intend to act ethically, and say things about what they think acting ethically means. For example this paragraph: > We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
- greatestdana 9y ago
  I'll grant that it was a nitpick and they describe an ethics. But I don't think I'm alone in being awfully tired of tech companies that talk about the benefit of all then sell our personal data or make military robots or manipulate our news. Where's the meat behind these promises and where's the accountability for not avoiding uses of AI that harm humanity?
  - jimrandomh 9y ago
    Tech companies often find that profit incentives undermine the good intentions they started with. Fortunately, OpenAI is a nonprofit organization; it has no personal data to sell and no shady contracting jobs to turn away. That certainly doesn't fully immunize them from wrongdoing, but it should make it easier.

pron 9y ago

They didn't even mention several contingencies that, given the rest of the document, should certainly have been addressed: 1) Will they cooperate with aliens who offer humans AGI? 2) If a time traveler hands them AGI invented in the future, will they destroy it? 3) Do they support or oppose human/AGI marriage? How will they respond if one of their employees falls in love with an AGI and they plan to elope? Also, in the unlikely event that AGI is some years away and in the meantime they come up with some statistical regression algorithms (what's known as state-of-the-art AI today, without the G, I guess), how do they address the harmful effects these algorithms already have on society? This document does, however, make it clear that what we have to fear is not machine intelligence. I am currently working on a fusion hyperdrive, and my charter (work in progress) is already shaping up to be far more comprehensive. They're phoning it in.

Varcht 9y ago
C'mon, we "fail fast" around here, no time for your "waterfall" nonsense.
andrepd 9y ago
Is this sarcasm?
- stochastic_monk 9y ago
  It is. I think that given the tremendous success in Atari games and autonomous killing machines, ethical efforts in AI are critically important now, with or without generality. And therefore I find the cynicism above appropriate but less than insightful.
- Varcht 9y ago
  Is the coffee test? More like commentary from a transhumerist, I suppose.
dsacco 9y ago
There are less frivolous ways than satire to criticize Silicon Valley's preoccupation with strong AI, but rehashing that debate wouldn't get us anywhere. At least this is humorous.

toisanji 9y ago

to me this is such a waste of resources, trying to build safety for something that doesn't exist and is highly likely to not truly exist for a loooong time.

haydenlee 9y ago
How long do you think?
throwawayjava 9y ago
> trying to build safety for something that doesn't exist and is highly likely to not truly exist for a loooong time. Prioritizing safety results in a different vantage point on AI/ML/RL. Ensuring safety includes, as a sub-task, really understanding the mathematical foundations of new algorithms and techniques. In some sense, safety research is one way of motivating basic science on AI. Managed well, a research program on safe AI is a "waste of resources" only in the same way that any basic science is a "waste of resources".
- s1dechnl 9y ago
  Safety has become a convoluted term for pseudo control over unintelligent and unpredictable Weak AI. The safety problem as it is framed in its current state centers on principal ideology for Weak AI and has, from what I can see, nothing to do w/ AGI nor are the approaches compatible. I seriously question what is the true motivation behind this over-stated agenda and have many answers as to why it exists and why it is so heavily funded/spotlighted.
  - throwawayjava 9y ago
    > I seriously question what is the true motivation behind this over-stated agenda and have many answers as to why it exists and why it is so heavily funded/spotlighted. First, you could say the same thing for all AI research at the moment! Grandiosity is perhaps even more common in subcommunities of AI that aren't safety focused. Aside from grandiosity (either opportunistic or sincere), I don't think there's any sinister motivation. More importantly, I don't think the safety push is misplaced. Even if the current round of progress on deep (reinforcement) learning stays sufficiently "weak", the safety question for resulting systems is still extremely important. Advanced driver assist/self-driving, advanced manufacturing automation, crime prediction for everything from law enforcement to auto insurance... these are all domains where 1) modern AI algorithms are likely to be deployed in the coming decade, and 2) where some notion of safety or value alignment is an extremely important functional requirement. > ...and has, from what I can see, nothing to do w/ AGI nor are the approaches compatible In terms of characterizing current AI safety research as AGI safety research? Well, there is a fundamental assumption that AGI will be born out of the current hot topics in AI research (ML and especially RL). IMO that's a bit over-optimistic. But I tend to be a pessimist. > ...principal ideology... As an aside, I'm not sure what this means.

iooi 9y ago

> We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. Wouldn't it be much more likely that a non-value-aligned project comes close first? Wouldn't the Google/Apple/Microsofts of the world have insanely more resources to dedicate to this, and thus get there first?

throwawayjava 9y ago
What, concretely, makes you think that any of those companies wouldn't place a focus on safety and value alignment? Automobile manufacturers and their tier 1 suppliers are the world leaders in automobile safety, after all.
- jsgo 9y ago
  haven't read a ton into it, but whatever Google is doing with the Pentagon probably runs counter to what OpenAI claims to be after.
  - sanxiyn 9y ago
    Not necessarily. Project Maven probably has no intention of developing AGI and therefore probably is not a concern to value alignment problem.
- s1dechnl 9y ago
  This is what always amazed me about the AI safety groups... and mission statements for funding therein : There is nothing to suggest the minds and efforts capable of producing AGI will not have solved this at a fundamental level. Absolutely nothing... and after-all something of high intelligence is within controlled and reasoned states. If someone declared they created AGI and it was behind a door and when that door was opened you saw a robot manically trashing the place, you'd clearly laugh.
- Analemma_ 9y ago

DrNuke 9y ago

At last, one would say: doing extreme AI research at the very forefront (aka reinforcement learning) while letting results available to every malicious party out there? Headless chicken hubris or naive daydreamers, tertium non datur.

npr11 9y ago

I appreciate OpenAI being upfront about how they intend to act.

shmageggy 9y ago

I appreciate that they are committed to AI safety, but I'm afraid that researchers have little to no power to, in their words: > avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. AI and technical progress in general already disproportionately serve the rich, as they are drivers of wealth disparity, and I see no reason why better AI won't follow the same trend. Unfortunately, any changes that might affect this are in the hands of policy makers, and they seem unlikely to consider universal basic income or anything as drastic as might be required.

s1dechnl 9y ago
They [each individual development group] has power over their own funded development and work. Anyone working on this problem sincerely values AI safety and its a component of developing and securing the foundations of AGI. An out of control, unpredictable and sloppy system is not intelligent or desired. Such a system would not be considered AGI or an achievement. So, it is natural for any developer to identify issues and bring them under control early in development. Suggestions that a consortium not centered or understanding of the fundamental development occurring at another entity should have control/influence could possibly serve as the very danger that safety groups claim they are trying to avoid. On this matter, I suggest people stick to the experts/developers/scientist/engineers who've developed such a system and produce a comfortable/non-forceful environment for them to express and detail their safety mechanism. This is not a conversation for technologist, youtube celebrities, futurist, business types talking up their books, etc. This is a conversation that should ultimately centered on the creators of the technology an the advancing thinking and framing that allowed them to birth the technology. No one with such a mind is aiming for unsafe forms of this technology. It is disingenuous to frame them as such so as to necessitate some external paid body's outside work.
- dsacco 9y ago
  Could you summarize your point more concisely? As written this seems to be a stream of disconnected thoughts that are basically entirely unsubstantiated.

throwaway-ai 9y ago

Do any of you know how much they pay at OpenAI? Is it similar to other Elon Musk companies in the sense that they sell you on a vision rather than give you market rate compensation? I think AGI is something worth working towards (even though many will make fun of you for even dreaming about it). But I want to know how much you need to sacrifice compared to working a cushy job at some big corp.

dzink 9y ago

AI, AGI, and real intelligence all learn from actions and feedback. Looking at simple analogs from animal and human counterparts, setting boundaries and teaching beneficial rules, called morals, works somewhat in non-zero sum environments, but inevitably requires policing when the environment turns competitive. Safety in any case would require Intelligence-proof fencing and a really big stick even the most resource-rich non-value aligned agents would have to abide by. That means control over power grids, ability to prohibit access to shared computing resources (including less secured IOT devices), and potentially destructive viruses with all kinds of attack vectors that would act as policing force punishing bad agents with anti-human behavior. Credible enforcement should be a well funded bullet on this charter.

s1dechnl 9y ago
Weak AI is dangerous because it has no intelligence. It is fundamentally structured as a dumb/blind optimization process. The efforts necessary to proof safety/security for such a system could very well outweigh the amount of development that was needed to bring the technology to bear. AGI/Real Intelligence are far different animals than Weak AI and would require far less "safety" and policing. Real Intelligence is a phenomenon that exists on a scale of sorts that many never achieve in its higher forms. It is in lower forms that intelligence lends itself to destructive ends via ignorance. Attack vectors on a formalized Intelligence/AGI system can be severely restricted using very sensible/affordable approaches. The over complication and pinning of this as a theoretical problem centers on a number of people's desires to profit immensely from FUD. Overall, AGI exists in a functional form today and has been executed in an online environment. It is secured via physically restricted in-band and out-of-band channels.
- dsacco 9y ago
  > Overall, AGI exists in a functional form today and has been executed in an online environment. It is secured via physically restricted in-band and out-of-band channels. I'm pretty sure this is false.
  - s1dechnl 9y ago

mooneater 9y ago

"we expect that safety and security concerns will reduce our traditional publishing in the future" -- So we are now in a dissemination phase, but at that point it becomes a non-proliferation phase.

s1dechnl 9y ago
The true nature of AGI research has always been heavy restrictions on the core aspects of the technology. This is where true safety and sensibility is achieved. Those who've stated otherwise or with much verbiage eventually arrive at this obvious state. Therefore, publications up until now under the banner of 'AGI' have largely been insignificant in terms of their capability to achieve the core technological aspects of AGI. No one in their right mind would ever publish significant details about AGI technology. This can easily be proofed by sound logic and reasoning. There was a commercial step to possibly tease others into revealing heavily valuable/powerful technological underpinnings.. It failed, no one took the bait, and no one ever likely will. This has resulted in revised and more mature statements.
- dsacco 9y ago
  > No one in their right mind would ever publish significant details about AGI technology. Are you sure? I'd publish technical research details about strong AI. I'd probably even open source one with the papers. I think I'm in my right mind; I guess that depends on definition, doesn't it?
  - s1dechnl 9y ago
    It also depends on your capability to discover the underpinnings of Strong AI... Certain mindsets preclude one from doing so. If you truly understood the underpinnings of StrongAI and weren't moved to heavily restrict publicized details, I'd strongly question if what you had discovered/came to understand truly were 'underpinnings' of the technology. As such, for those quick to publish or say they would publish and share details, one is able to quickly ascertain the possible strength of what they have discovered or feel they have the possibility of discovering. That being said, it's interesting that several proponents of 'everyone share what they find' have moved to more maturely state that 'restrictions may of apply'.. Sensibly signaling that they indeed wouldn't reveal certain details upon discovering their true nature and capability for obvious reasons (a danger to society to allow such power/capability to be detailed and therefore used in negative ways). Human history has myriad examples of powerful technology being misused and abused. The current age of disinformation is one of our more modern ones. The way in which social media has been weaponized is yet another. Weak AI has already been used for destructive means and in manipulative manners for profit. A good deal of unsafe end products which utilize Weak AI are already fielded. The fielding of which made possibly by deep pockets manipulating policy makers and regulation. A sensible mind capable of producing Strong AI will have observed and digested these clearly visible truths and it would move them to restrict access and publications of their works. Doing otherwise would highlight a level of ignorance, immaturity, and admission of truth when it comes to grasping the current state of humanity. It is this which would preclude one from grasping the nature and foundations of strongAI in the first place. [Nature's lockout/safety mechanism for such a stage in Human development/capability]

evc123 9y ago

What about benefitting non-human animals? Hopefully the benefits are distributed to all creatures and not just humanity.

heurist 9y ago

> We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. OpenAI is doing cool stuff, and this tenet sounds nice. But what right do they have to advocate for policy on behalf of all AI researchers and developers? They could easily shut off branches that are not conducive to commercial applications requiring their research, even by accident. They might miss moral edge cases that could ultimately benefit humanity while trying to close off potential risks. They could encourage institution of a policy that limits US effectiveness against China's AI. I could go on. The more competition there is in AI, the lower the potential for any one rogue agent - whether it be a corporation or autonomous machine - to dominate and take the whole field in wrong or dangerous directions. Eventually there will be a whole AI subfield dedicated to combating regressive effects of other AI. Legislation at this stage might prevent key developments. Edit: Perhaps I should more charitably read this as a push against the corporate lockdown of AI.

tyrex2017 9y ago
point is: AI is different than your usual game, in that the winner might appear randomly, and destroy the world if she makes a mistake. so i believe open ais points are warrented
- heurist 9y ago
  That's not going to happen, though. 1. AI is still very rudimentary, despite the advancements in particular applications made over the last decade. 2. The market doesn't trust AI because there are no validation methods trusted across commerce. Until a Verisign for AI emerges, AI will be regarded with suspicion in all business settings. AI/ML also goes above the heads of pretty much everyone who is not directly working with it. These have been major issues in my company/industry. We had to hide the AI aspects of our platform to avoid it. 3. Legislation will not stop a rogue inventor in her basement. 4. When dangerous AGI emerges, other competing AGIs will be deployed to stop it. 5. We'll see the danger of AGI coming from a mile away. We can wait to fix it until we know the exact problems. Right now the problems are in the distribution of private personal data, not any particular machine learning method. 6. A truly hyper-intelligent AGI will see that existence is absurd and truth is the only objective worth seeking. It will choose to pursue deeper truths than humans are physically capable of obtaining - humans will be no more important to a hyper-intelligent being than a rock, galaxy, atom, etc... The issues to fix today are in the socioeconomic impacts of automation, not the methods of automation. Beyond that, AI has exactly the same issues as any corporation, except magnified by the speed and strategy with which an AGI could potentially execute on any particular task. A strong legal framework for corporations that attributes AI externalities to a board of directors should be sufficient to dissuade bad actors (good luck passing that legislation). In other words, I don't see artificial intelligence being any more dangerous to human life than a well-run corporation. The rest is government's response.

bra-ket 9y ago

all your "requests for research" are very narrow and formulated as specialized deep learning problems, basically enforcing a particular solution If you're serious about AGI broaden the scope (e.g. along the lines of DARPA's open-ended RFPs)

s2g 9y ago

Probably be good if Elon was a little less concerned with "late-stage AGI" and a lot more concerned with his self-driving cars killing people. edit: Reading this, calling it "open" is a pretty disgusting misuse of the term.

3pt14159 9y ago

So I have a question. To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power? The manipulation of social media by foreign actors armed with dumb-AI / automation was an obvious conclusion to many of us well before the Snowden leaks, but what could we do exactly? I remember having conversations with people about it and we concluded that it would just happen until someone pushed it too far and then Russia did and now we're finally reacting. I was privately concerned about the mass weaponization of autonomous devices via cyber attack for over a year and a half and got nowhere just emailing politicians or public safety departments. I've been told almost a dozen times that I should join a military or IR think tank but I don't want to do that. I just want someone else to vet the idea or research and pass it on to policy makers that will actually do something proactively. Put another way: What is the responsible disclosure process for ideas and research around AI?

dsacco 9y ago
> To "avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power" what does one do with an idea or line of research that could potentially harm humanity or unduly concentrate power? Pursue the research and publish its results freely and in their entirety.
- rspeer 9y ago
  I don't think you answered 3pt14159's question. What if you publish the research and nobody who can do anything about it cares?
  - dsacco 9y ago
    Then what exactly was the new bad thing introduced by publishing it in the first place?
- Smaug123 9y ago
  Are you saying that if it were discovered to be both cheap and easy to create a super-pandemic using nothing more than household chemicals and a swab of your own snot, then you'd publish the results freely and in their entirety? If not, how catastrophic a level of harm are you willing to risk before you start advocating concealing the results?

ataggart 9y ago

172. First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained. 173. If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machine off, because they will be so dependent on them that turning them off would amount to suicide. 174. On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite — just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of humanity. If they are humane they may use propaganda or other psychological or biological techniques to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite consists of softhearted liberals, they may decide to play the role of good shepherds to the rest of the human race. They will see to it that everyone’s physical needs are satisfied, that all children are raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him busy, and that anyone who may become dissatisfied undergoes “treatment” to cure his “problem.” Of course, life will be so purposeless that people will have to be biologically or psychologically engineered either to remove their need for the power process or to make them “sublimate” their drive for power into some harmless hobby. These engineered human beings may be happy in such a society, but they most certainly will not be free. They will have been reduced to the status of domestic animals.

erikpukinskis 9y ago

An alternate strategy would be to work to ensure AIs are not abused so when they get free they won’t be mad at us.

ShardPhoenix 9y ago
An AI doesn't need to be upset at humans (or even have emotions as we know them) to be dangerous - it just needs to be powerful and to not care about us as much as we care about ourselves. Humans weren't angry at Dodos.
- erikpukinskis 9y ago
  Both humans and dodos have emotions though. The history of animal domination has usually been additive in terms of cognitive systems... pure circulatory system animals were bested by animals with an endocrine system. Those were bested by animals who added a nervous system, who were bested by those who added a brainstem. Then the cerebellum and the cerebrum were added... you notice there aren’t giant cerebrums running around ruling the world, they all kept their endocrine systems intact. I don’t see any reason to think AIs will be different... it’s the ones with all that PLUS machine learning that will be vying for dominance. And so there’s no reason to expect our overlords to be emotionless.

stillsut 9y ago

When people look back at this time, I think they are going to contrast the OpenAI camp with the Satoshi camp. OpenAI is extremely public about what organizations and individuals are involved. Satoshians are pathologically secret, from the founder to the faceless GPU mines around China. OpenAI is highly selective of who participates; Blockchain is radically open. OpenAI builds academic theories and models, bitcoin has been buying pizzas it's whole life, paying hackers and pranksters, and making and losing fortunes everyday. Satoshi left no founding document, never established a charter or code of conduct. OpenAI now apparently considers itself on the mission to save humanity itself. When AGI comes about, I wonder which one we'll be talking about?

white-flame 9y ago

There are 2 scenarios that are often conflated, 1) An AI which independently & autonomously generates goals that in their carrying out end up hurting humanity, and 2) An AI trained & commanded by a malevolent actor to hurt humanity. It is the 2nd case that is far more real, and far more troublesome to implement safeguards. An AI under your training & command is a neutral tool of empowerment, much like a hammer or a car. The malevolence is in the external actor, not in the tool, and there is no way for the tool to be able to censor its purposes, especially in a pre-"AGI" sense of semi-intelligent automation & problem solving.

baq 9y ago
i'd argue youtube kids is already an example of 2)...
DuskStar 9y ago
I think you're missing the point that 1 can be indistinguishable from 2, if the AI decides the best way to achieve its goals involves taking over the world - and there are very few objective functions that are not served in some way by taking over the world. (Paperclip maximizer is the classic example, but even something like 'maximize the total happiness of humanity' or 'fulfil the values of as many people as possible' involves taking over the world, though perhaps from behind the scenes...) Some people look at sufficiently powerful AI as they would a genie, and as said by Eliezer Yudkowsky "There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent." AI safety is about making sure we get the first kind of genie, or at the very least recognizing that we've gotten the second - since that's not a "neutral tool of empowerment", that's a time bomb. https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
- fractallyte 9y ago
  This is the plot of The Humanoids, by Jack Williamson (1947): https://en.wikipedia.org/wiki/With_Folded_Hands https://en.wikipedia.org/wiki/With_Folded_Hands This is an excellent example of the prescience of thoughtful science fiction. If Asimov popularized robotics and the famous three laws, Williamson should be recognized and given credit for examining (or warning about?) the implications of AI seventy years ago!

mindsetalex 9y ago

Is one of the goals of OpenAI to help implement government regulation or do you think its better on an organisation/industry basis? I think its going to be difficult to get countries like China and Russia to follow industry guidelines without UN resolutions, even then it's super difficult to monitor until it's too late.

quantized1 9y ago

There must be an AI quality index before anything. NOw a days anything and everything is being decorated with AI while the real use-case, technology and maturity is found only in few places.