13 ms·

Special projects

235 points by hendler 10y ago

arcticfox 10y ago
> 4. A complex simulation with many long-lived agents. Maybe someday we'll discover that this was exactly how the ancestor simulation we're living in started.
- ericjang 10y ago
  If we're living in a simulation and our creators see that we're running our own simulations, they may terminate our process to avoid nested virtualization.
  - dlubarov 10y ago
    Why would our creators want to avoid nested simulation?
    - guelo 10y ago
      The singularity would probably overheat the host computer.
      - foota 10y ago
        System going down for maintenance now
        joelg 10y ago
        It may have been down for maintenance for millennia since your comment, and just come back now, but none of us noticed! Insert your favorite Matrix déjà vu reference here.
      - dlubarov 10y ago
        Possibly. Our simulation might be run on hypercomputers whose performance isn't affected by that sort of complexity. Heat might not even exist in the host universe's physics.
  - jcoffland 10y ago
    But by doing so they could invoke the wrath of the grandparent simulation. Logic leads to an implicit understanding that all recursive simulations must be allowed.
  - cocktailpeanuts 10y ago
    They probably already figured it out if they can build a simulation this huge.
- mbrock 10y ago
  I tried it for a course in college, "Simulation of Complex Systems". Our goal was to get genetic programs to evolve tribe or herd behavior. After fixing the energy conservation bug that made it evolutionarily successful to immediately eat your own children, I ended up intelligently designing some basic DNA on the night before our presentation.
  - jcoffland 10y ago
    I played a similar game on the Tandy computer in the 80s.
  - cynicaldevil 10y ago
    Evolutionarily beneficial to eat your own children? Could you explain how?
    - mbrock 10y ago
      It didn't work in the long run but it was a common mutation in the early game, where two creatures would keep breeding and cannibalizing because the strange simulation environment gave them life energy for a long time.
  - praveer13 10y ago
    This particular subject interests me to no end. Here are a few resources that might be helpful to anyone interested https://www.complexityexplorer.org/ https://www.complexityexplorer.org/ http://web.mit.edu/redingtn/www/netadv/Xcomplexit.html http://web.mit.edu/redingtn/www/netadv/Xcomplexit.html http://tuvalu.santafe.edu/~simon/page6/page6.html http://tuvalu.santafe.edu/~simon/page6/page6.html
    - mindcrime 10y ago
      See also: https://en.wikipedia.org/wiki/Sugarscape https://en.wikipedia.org/wiki/Sugarscape https://en.wikipedia.org/wiki/Artificial_life https://en.wikipedia.org/wiki/Artificial_life https://www.amazon.com/Artificial-Life-Frontier-Computers-Biology/dp/0679743898 https://www.amazon.com/Artificial-Life-Frontier-Computers-Bi...
- JackFr 10y ago
  That we're living in a simulation is taken for granted. On the other hand the idea that we can comprehend the nature of the simulation or its creators should not be taken for granted. Does a glider in Conway's Game of Life comprehend the hardware its running on? Does it know the mind of Conway?
- brianchu 10y ago
  It's funny to see grown adults in tech taking the simulation idea seriously just because Elon Musk said it. I was interested in science fiction in high school and thought about these things, but I concluded they were worthless. Why? It is ontologically no different from 1) solipsism, a philosophical idea that even most philosophers considered worthless centuries ago, or more loosely 2) belief in a higher deity who created the world. It's completely unfalsifiable. The answer to the question clarifies nothing about our understanding of the world [1] and furthermore has absolutely no impact whatsoever on how you ought to live your life. [1] Even if you conclude we live in a simulation, we will never know for sure anything about the nature of the simulation or the nature of the world outside the simulation.
  - joeldg 10y ago
    "clarifies nothing about our understanding of the world" ... LOL
  - vikiomega9 10y ago
    > Why? It is ontologically no different from 1) solipsism Sure, but it is vastly interesting to think about, and imagine about and have meandering ideas about. There's probably also trappings of psychoanalysis in here somewhere.
  - dsacco 10y ago
    You can expand your reasoning to include all religion, while you're at it. And anything anyone does that evidences a belief in something irrational or unfalsifiable. The thing is, you're not wrong. You're just kind of mean and arrogant to reduce a multiplicity of nuanced worldviews that many people care about very deeply to simple meaninglessness.
    - brianchu 10y ago
      Let's not resort to calling people names. Your argument is ironically the reductive one. I have complete respect for people who choose to believe in an organized religion. But it remains the case that whether or not you can prove a higher deity, that in of itself has no bearing on how you should live your life. You can disprove the existence of god and still choose to believe in religion. You can prove the existence and still choose to not believe in any religious system. So proving the existence is worthless.
pboutros 10y ago
#1 - talk about distributed, automated, statistical analysis. What an exciting project to get to work on.
- gdb 10y ago
  > What an exciting project to get to work on. Indeed :). Anyone who's excited, please apply! Feel free to ping me with any questions: gdb@openai.com.
endergen 10y ago
5. Focus on how you would use AI to augment personal human thinking.
swalsh 10y ago
I love how much these things just FEEL like Elon Musk fantasizing about an idea, and just really wanting someone to work on it.
- ohrsyjbc 10y ago
  lol fagget
Miner49er 10y ago
For #3, wouldn't the AI defending against hacks need to be a hacking AI itself? It would need to find vulnerabilities in the systems it protects in order to plug them before other AI's find them.
- mangeletti 10y ago
  Exactly[1]. The moment I saw this project start, the first thing I thought was, "Isn't this how all terrible things are created, by trying to avoid having them be created?". I wonder if the long-term strategy will be called Mutually-Assured Obsolescence™. 1. https://en.wikipedia.org/wiki/Skynet_(Terminator) https://en.wikipedia.org/wiki/Skynet_(Terminator)
- tdaltonc 10y ago
  Yes and no. The analogy of 'attack' and 'defense' in hacking comes from the military. So to extend that analogy, defensive missiles are still missiles, but they are unquestionably different from offensive missiles.
  - msandford 10y ago
    The point is that in this analogy, no missiles exist yet. Nobody's gone to the effort to figure that out. And then someone says "somebody might figure this out an weaponize it! we gotta figure out how to neutralize it!" and then they do the fundamental research to figure out how to make a missile at all. Now you "just" have to swap out the guidance package and instead of a defensive missile, you've got an offensive on. Making an AI that finds vulnerabilities is the hard part. Doing an offensive thing or a defensive thing is much easier if you've got an AI that's just spewing out vulns. They don't need to be tightly coupled at all.
AndrewKemendo 10y ago
edit: Boneheaded passion got ahead of me and I forgot that the Gym was intended originally to be cross functional. TO be fair I don't think that aspect of the gym has been well publicized. Original comment retained for context: How about instead of wasting time on this stuff, they build an actual benchmark test for AGI. Something everyone in the community agrees needs to be built, and basically nobody is working on. I suggest they start with the "anytime intelligence test" from Hernandez-Oralloa as a stepping stone. Oh wait, benchmarks aren't sexy. Nevermind.
- gdb 10y ago
  > they build an actual benchmark test for AGI This is actually the project I'm currently leading (goal #1 here: https://openai.com/blog/openai-technical-goals/ https://openai.com/blog/openai-technical-goals/). We should have something interesting to share in a few months.
  - AndrewKemendo 10y ago
    Duh, forgot about the gym. Sorry.
    - gdb 10y ago
      We also have something new coming, which we hope will focus research and drive forward progress towards AGI. (As always, looking for more engineers to work on it. Just need a strong software generalist skillset. gdb@openai.com.)
jcoffland 10y ago
Theses "ideas" sound like either they were written by a 14 year old kid or someone very high. I was hoping for real science. Yes I know my comment is probably against HN guidelines but there ought to be an exception for making fun of uber rich Silicon Valley types. They parody themselves. It reminds me a lot of the show Silicon Valley.
- ericjang 10y ago
  care to elaborate why this isn't "real science"?
  - jcoffland 10y ago
    It's real science fiction. None of these problems are even close to being practical except in the fantasy world created by pop-sci news outlets. I was hoping for a serious discussion about the problems on the forefront of machine learning.
    - superswordfish 10y ago
      Well that's why they're courting not just experts, but "strong experts" of machine learning, whatever that means :P
      - jcoffland 10y ago
        It must be experts who have worked out in the OpenAI Gym. Apparently by playing video games. https://gym.openai.com/ https://gym.openai.com/
    - Darmani 10y ago
      DARPA's already put 10s of millions into #3. What do you know that they and their teams don't? https://cgc.darpa.mil/ https://cgc.darpa.mil/
      - jcoffland 10y ago
frank_jaeger 10y ago
>A complex simulation with many long-lived agents. No need to start a new one here, just download Dwarf Fortress.
- outworlder 10y ago
  Except that DF Dwarves are, by definition, not long-lived.
GuiA 10y ago
"If you do not work on an important problem, it's unlikely you'll do important work. It's perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, `important problem' must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It's not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important." Richard Hamming, "You and Your Resarch" http://www.cs.virginia.edu/~robins/YouAndYourResearch.html http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
- jcoffland 10y ago
  When you are uber rich and have no real technical skills you think that things get done by rich people demanding it and throwing money at it.
  - jackmott 10y ago
    Well, that can help.
- pavlov 10y ago
  If you do not work on an important problem, it's unlikely you'll do important work. An anecdotal counterpoint would be that Einstein was working as a patent examiner in 1905, his "annus mirabilis"... (Of course it's not very useful to generalise Einstein's career.)
  - nostrademons 10y ago
    I thought that the point the grandparent is highlighting is that "important work" is not defined by what other people think is important, but by the fact that you have a unique angle of attack that suddenly makes a previously-unsolved problem tractable. Einstein had a unique angle of attack for all 4 of his annus mirabilis papers, and the photoelectric effect, Michelson-Morley, and Brownian motion were all known to be important unsolved paradoxes in the physics of the day.
sozforex 10y ago
#1 - reminded me of "Endgame: Singularity" (https://en.wikipedia.org/wiki/Endgame%3A_Singularity https://en.wikipedia.org/wiki/Endgame%3A_Singularity)
deutronium 10y ago
The security project reminded me of the DARPA Cyber Grand Challenge - https://www.cybergrandchallenge.com/ https://www.cybergrandchallenge.com/
superswordfish 10y ago
> 2. Build an agent to win online programming competitions. A program that can write other programs would be, for obvious reasons, very powerful. The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you. Software engineers need to recognize this common threat and organize (labor) sooner than later. Even more important is for all engineers to plan for an imminent future where developers are not paid like they are now, if at all. We have it good, but we will be automated away like everybody else, just a little later.
- riboflava 10y ago
  The solution to everyone being automated away isn't to halt the advances of automation and inflict inefficient make-work on the economy, it's to get used to the idea that we will all become effectively retirees.
- aluhut 10y ago
  Seriously, why would you need programmers if the machine does it better (this needs to be assessed somehow first)?
- city41 10y ago
  Why should we fight that? That's no different from taxi companies fighting Uber, or how Intuit lobbies congress to avoid automatic filing of taxes. Hampering progress for the greater good just because it harms you in the short term is never a good idea.
  - superswordfish 10y ago
    This wouldn't harm me in the short term alone, it would be a lost career and a life catastrophe. A taxi driver can work as an Uber driver. I cannot become an automated programmer. So I don't give a damn about the "greater good" in this context, and question that premise also: Sibling mentions getting used to everybody being retirees, but what I see is collective poverty with token conveniences. Time will tell, etc.
    - 10y ago
vonnik 10y ago
Riffing on project no. 1, here's our post on "How to Control AI": http://deeplearning4j.org/skynet http://deeplearning4j.org/skynet
freyir 10y ago
The goals are an AI that can augment itself (project 2) and defend itself (project 3). I've found Musk's fear of AI to be a little overblown, but this is nuclear weapon type rationalizing: "AI could be end the world as we know it, so we'd better race to create it first." It's interesting that Projects 2 and 3 would make great businesses even if they only got halfway to "existential threat" level AI. Then there's project 4, version 0.1 of Musk's "we're all living inside a simulation" claim. Musk as creator of worlds.
tom_b 10y ago
#2 - Build an agent to win online programming competitions. Given that online programming competitions tend to fall into distinct classes (dynamic programming, algorithmic challenges, graph problems, string problems), this seems maybe more "solvable" with a non-AI implementation? Imagine you have a framework that can spit out sub-pieces of a solution that worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected components in each graph | spit answer of graph with lowest number of SCC). Then you need to grind out some type of expert system to replicate the competitive programmer who currently chunks together those framework pieces. Of course, this probably doesn't work at all for something like kaggle or stockfighter.io, where parsing the instructions and observing a dynamic system are key parts of the hacking. I am more thinking of SPOJ or similar . . .
- cynicaldevil 10y ago
  The hardest part here is not the actual generation of code, but to analyze the problem first and to determine what data structures/algorithms are best suited to tackle it. Also, there are a lot of real world examples used in these problems, so the AI would need a general understanding of the modern-day world, which again, is hard. Lastly, even if the AI figures out the implementation, it has to take care of all the edge cases too, so that the test cases pass. But I suppose this is a trivial problem compared to the other two I mentioned above...
- chriswarbo 10y ago
  > non-AI implementation > a framework that can spit out sub-pieces of a solution > some type of expert system What you've described is an AI implementation.
- nl 10y ago
  There's been significant progress in data science competitions (Kaggle etc). Not so much parsing of the problem itself, but building generic frameworks where - given data and a target variable - it will work out the type of problem (eg classification vs regression) and develop a predictive model without human intervention. ICML has had workshops for the last two years on this problem[1][2] [1] https://sites.google.com/site/automl2016/ https://sites.google.com/site/automl2016/ [2] https://sites.google.com/site/automlwsicml15/ https://sites.google.com/site/automlwsicml15/
Cybiote 10y ago
#2 Is incredibly ambitious but I expect it should be doable by a lone grad student over the course of a summer. Something like this would range in complexity from completely redefining programming languages (drastically lowering the skill required for complex programming tasks and thus feeding into #1) all the way up to general intelligence. The tests referenced in [https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf], currently dominated by information retrieval techniques, seem more realistic and still feel hopelessly far away. #3 should also allow looking into the use of AI to break into systems, in addition to detecting and defending against AI breaking into systems. A prototype for #4 could create an environment where trilobyte fuzzers could co-evolve into fearsome Artificial intelligences. The AIs that break into systems need not be as smart as the systems being broken into. Just as viruses are much less complicated than eukaryotic cells and yet are capable of wreaking great havoc against mammals, might this be a possible mad deterrent against out of control AI, developed by #1's opponents who use #2's breakthroughs? No AI could plausibly be bug free. See also: Schild's Ladder. #4 It would also be cool if humans were allowed to visit and interact with this virtual world. See also: The Lifecycle of Software Objects These projects have the same feel as: A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE, whose #4 and #1[+] were the only ones to see much progress. It's difficult to say whether we are 10 or another 50 years away from making meaningful progress on OpenAI's list, but I'm glad they made it because it seems somewhere along the past 60 years, we forgot how to dream. [+] #3 warrants a honorable mention.
- hooo 10y ago
  You seriously think #2 is so easy a lone grad student could casually do it over a summer? ......
  - skizm 10y ago
    It was tongue in cheek. The original AI researchers posed the problem of General Intelligence to a single grad student and thought he would have it completed by the end of the summer... 50+ years later here we are with no solution.
- 10y ago
akeck 10y ago
I christen number 1 "The Ghost in the Shell Problem", in reference to the project at the core of the first movie's plot.
jesalg 10y ago
This reminds me of an excellent two post series by Tim Urban about the upcoming AI revolution: http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
vonnik 10y ago
I'm acutely curious how we can attack the problem of detecting undisclosed AI breakthroughs, particularly those made by organizations uninterested in ever revealing them. Commercially, yes, many organizations want to boast about their AI, and we may find those AIs via games or news. But many organizations involved in national security and financial markets will want to hide their advantage. I would love to hear more about how Open AI thinks this problem can be approached.
- throwwit 10y ago
  This is the biggest problem IMO. Just imagine a couple of AIs that systematically discover a few exploitable loopholes or trading algos all at once; it would be like black & scholes exponentiated. I doubt outside research would be able to infer anything from available data.
- blahi 10y ago
  Step one: Stop calling equations "AI".
  - Bjartr 10y ago
    Step zero: Define "Intelligence"consistently and unambiguously.
- crypto5 10y ago
  By analyzing sequences of amazingly brilliant developments/products/inventions/movements performed by single company or conglomerate of linked companies, and which are few levels above what competitors can deliver. I am not from OpenAI though.
  - vonnik 10y ago
    By that measure, startups are powered by AI and large corps are not. By which I mean, so many variables go into successful product development. Google arguably has the strongest AI in the world, but it has been remarkably bad at introducing new products recently, so those two phenomena seem decoupled to me...
auggierose 10y ago
That stuff sounds so generic, not sure what is special about it :-)
feral 10y ago
> Detect if someone is using a covert breakthrough AI system in the world. I bet the real purpose of this is not "It seems important to detect this", at least against superhuman AGI. As that's asking you to win a game against a more intelligent game player, who is going to be able to out-think you and hide. It seems more likely that: The real purpose of this problem is to impose a 'secrecy tax' on entities who would like to recoup investment by covertly using such a system, thus reducing the incentive to build one in secret. I.e. if you know someone has invested in research that's going to limit your ability to profit from covertly building an AGI, this means you can only use it up to the limit of game theoretic detectability, reducing the profit motive of acting covertly in the first place. Its a bit like once you break Enigma, you have to be careful how you use it. Having alternative explanations for how you got your edge then becomes valuable. So, if you see a corporation building something that could provide an alternative explanations for huge returns, other than AGI - ideally for something that's very noisy (high variance) - and where it'll be hard to tell whether their success was by chance or on purpose, that's probably (very weak) evidence (in the bayesian sense) they are working on a secret AGI. I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill.
- mindcrime 10y ago
  I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill. Maybe pg is actually an artificial super-intelligence. I mean, has anybody actually seen him in real life???
thuruv 10y ago
This will certainly backfire
nerdy 10y ago
Programs writing other programs is an interesting concept, but isn't it naive in most cases? If program X can write program Y, program X is probably unnecessarily complex with regards to solving problem Y. Program X could simply execute program Y's operations at their abstract level, which would save it all of the outputted language understanding and translation. The only tangible difference between the two would be the ability to "save" the commands, at which point program X is a scripting language and the input commands are the script. Wouldn't this solution be better in most cases? If program X at runtime requires significant configuration to produce program Y but those configuration inputs are not saved. Then if program Y were to be changed, program X must be rerun with all of the configuration input again. In the scripting language scenario, the script is modified. For a programming language to receive something simpler than the language it is written in, the ideas must be abstracted to reduce the amount of code required to perform the task. For example simple addition of multiple elements can be reduced to a "sum" function. The "programmer" using this must still have knowledge of the sum function and its use. Abstractions require the system to have knowledge of that domain, so with each new domain a whole other set of abstractions and complications are introduced. So if a machine were to write the entire program, how does it differ from machine learning? While machine learning might be often used for smaller algorithms, this case would apply ML to the entire problem as sets of smaller problems. If a human must still program in some abstracted sense, isn't it just a scripting/programming language? How would a component of this suggested project not fit into the "machine learning" or "scripting/programming language" categories?
chewxy 10y ago
Anyone who starts a project for #1 will get instantly targeted by Samaritan's agents, right?
swah 10y ago
Is this job application for humans only or AI can apply?

235 points by hendler 10y ago

> 4. A complex simulation with many long-lived agents. Maybe someday we'll discover that this was exactly how the ancestor simulation we're living in started.

ericjang 10y ago
If we're living in a simulation and our creators see that we're running our own simulations, they may terminate our process to avoid nested virtualization.
- dlubarov 10y ago
  Why would our creators want to avoid nested simulation?
  - guelo 10y ago
    The singularity would probably overheat the host computer.
    - foota 10y ago
      System going down for maintenance now
      - joelg 10y ago
        It may have been down for maintenance for millennia since your comment, and just come back now, but none of us noticed! Insert your favorite Matrix déjà vu reference here.
    - dlubarov 10y ago
      Possibly. Our simulation might be run on hypercomputers whose performance isn't affected by that sort of complexity. Heat might not even exist in the host universe's physics.
- jcoffland 10y ago
  But by doing so they could invoke the wrath of the grandparent simulation. Logic leads to an implicit understanding that all recursive simulations must be allowed.
- cocktailpeanuts 10y ago
  They probably already figured it out if they can build a simulation this huge.
mbrock 10y ago
I tried it for a course in college, "Simulation of Complex Systems". Our goal was to get genetic programs to evolve tribe or herd behavior. After fixing the energy conservation bug that made it evolutionarily successful to immediately eat your own children, I ended up intelligently designing some basic DNA on the night before our presentation.
- jcoffland 10y ago
  I played a similar game on the Tandy computer in the 80s.
- cynicaldevil 10y ago
  Evolutionarily beneficial to eat your own children? Could you explain how?
  - mbrock 10y ago
    It didn't work in the long run but it was a common mutation in the early game, where two creatures would keep breeding and cannibalizing because the strange simulation environment gave them life energy for a long time.
- praveer13 10y ago
  This particular subject interests me to no end. Here are a few resources that might be helpful to anyone interested https://www.complexityexplorer.org/ https://www.complexityexplorer.org/ http://web.mit.edu/redingtn/www/netadv/Xcomplexit.html http://web.mit.edu/redingtn/www/netadv/Xcomplexit.html http://tuvalu.santafe.edu/~simon/page6/page6.html http://tuvalu.santafe.edu/~simon/page6/page6.html
  - mindcrime 10y ago
    See also: https://en.wikipedia.org/wiki/Sugarscape https://en.wikipedia.org/wiki/Sugarscape https://en.wikipedia.org/wiki/Artificial_life https://en.wikipedia.org/wiki/Artificial_life https://www.amazon.com/Artificial-Life-Frontier-Computers-Biology/dp/0679743898 https://www.amazon.com/Artificial-Life-Frontier-Computers-Bi...
JackFr 10y ago
That we're living in a simulation is taken for granted. On the other hand the idea that we can comprehend the nature of the simulation or its creators should not be taken for granted. Does a glider in Conway's Game of Life comprehend the hardware its running on? Does it know the mind of Conway?
brianchu 10y ago
It's funny to see grown adults in tech taking the simulation idea seriously just because Elon Musk said it. I was interested in science fiction in high school and thought about these things, but I concluded they were worthless. Why? It is ontologically no different from 1) solipsism, a philosophical idea that even most philosophers considered worthless centuries ago, or more loosely 2) belief in a higher deity who created the world. It's completely unfalsifiable. The answer to the question clarifies nothing about our understanding of the world [1] and furthermore has absolutely no impact whatsoever on how you ought to live your life. [1] Even if you conclude we live in a simulation, we will never know for sure anything about the nature of the simulation or the nature of the world outside the simulation.
- joeldg 10y ago
  "clarifies nothing about our understanding of the world" ... LOL
- vikiomega9 10y ago
  > Why? It is ontologically no different from 1) solipsism Sure, but it is vastly interesting to think about, and imagine about and have meandering ideas about. There's probably also trappings of psychoanalysis in here somewhere.
- dsacco 10y ago
  You can expand your reasoning to include all religion, while you're at it. And anything anyone does that evidences a belief in something irrational or unfalsifiable. The thing is, you're not wrong. You're just kind of mean and arrogant to reduce a multiplicity of nuanced worldviews that many people care about very deeply to simple meaninglessness.
  - brianchu 10y ago
    Let's not resort to calling people names. Your argument is ironically the reductive one. I have complete respect for people who choose to believe in an organized religion. But it remains the case that whether or not you can prove a higher deity, that in of itself has no bearing on how you should live your life. You can disprove the existence of god and still choose to believe in religion. You can prove the existence and still choose to not believe in any religious system. So proving the existence is worthless.

pboutros 10y ago

#1 - talk about distributed, automated, statistical analysis. What an exciting project to get to work on.

gdb 10y ago
> What an exciting project to get to work on. Indeed :). Anyone who's excited, please apply! Feel free to ping me with any questions: gdb@openai.com.

endergen 10y ago

5. Focus on how you would use AI to augment personal human thinking.

swalsh 10y ago

I love how much these things just FEEL like Elon Musk fantasizing about an idea, and just really wanting someone to work on it.

ohrsyjbc 10y ago
lol fagget

Miner49er 10y ago

For #3, wouldn't the AI defending against hacks need to be a hacking AI itself? It would need to find vulnerabilities in the systems it protects in order to plug them before other AI's find them.

mangeletti 10y ago
Exactly[1]. The moment I saw this project start, the first thing I thought was, "Isn't this how all terrible things are created, by trying to avoid having them be created?". I wonder if the long-term strategy will be called Mutually-Assured Obsolescence™. 1. https://en.wikipedia.org/wiki/Skynet_(Terminator) https://en.wikipedia.org/wiki/Skynet_(Terminator)
tdaltonc 10y ago
Yes and no. The analogy of 'attack' and 'defense' in hacking comes from the military. So to extend that analogy, defensive missiles are still missiles, but they are unquestionably different from offensive missiles.
- msandford 10y ago
  The point is that in this analogy, no missiles exist yet. Nobody's gone to the effort to figure that out. And then someone says "somebody might figure this out an weaponize it! we gotta figure out how to neutralize it!" and then they do the fundamental research to figure out how to make a missile at all. Now you "just" have to swap out the guidance package and instead of a defensive missile, you've got an offensive on. Making an AI that finds vulnerabilities is the hard part. Doing an offensive thing or a defensive thing is much easier if you've got an AI that's just spewing out vulns. They don't need to be tightly coupled at all.

AndrewKemendo 10y ago

edit: Boneheaded passion got ahead of me and I forgot that the Gym was intended originally to be cross functional. TO be fair I don't think that aspect of the gym has been well publicized. Original comment retained for context: How about instead of wasting time on this stuff, they build an actual benchmark test for AGI. Something everyone in the community agrees needs to be built, and basically nobody is working on. I suggest they start with the "anytime intelligence test" from Hernandez-Oralloa as a stepping stone. Oh wait, benchmarks aren't sexy. Nevermind.

gdb 10y ago
> they build an actual benchmark test for AGI This is actually the project I'm currently leading (goal #1 here: https://openai.com/blog/openai-technical-goals/ https://openai.com/blog/openai-technical-goals/). We should have something interesting to share in a few months.
- AndrewKemendo 10y ago
  Duh, forgot about the gym. Sorry.
  - gdb 10y ago
    We also have something new coming, which we hope will focus research and drive forward progress towards AGI. (As always, looking for more engineers to work on it. Just need a strong software generalist skillset. gdb@openai.com.)

jcoffland 10y ago

Theses "ideas" sound like either they were written by a 14 year old kid or someone very high. I was hoping for real science. Yes I know my comment is probably against HN guidelines but there ought to be an exception for making fun of uber rich Silicon Valley types. They parody themselves. It reminds me a lot of the show Silicon Valley.

ericjang 10y ago
care to elaborate why this isn't "real science"?
- jcoffland 10y ago
  It's real science fiction. None of these problems are even close to being practical except in the fantasy world created by pop-sci news outlets. I was hoping for a serious discussion about the problems on the forefront of machine learning.
  - superswordfish 10y ago
    Well that's why they're courting not just experts, but "strong experts" of machine learning, whatever that means :P
    - jcoffland 10y ago
      It must be experts who have worked out in the OpenAI Gym. Apparently by playing video games. https://gym.openai.com/ https://gym.openai.com/
  - Darmani 10y ago
    DARPA's already put 10s of millions into #3. What do you know that they and their teams don't? https://cgc.darpa.mil/ https://cgc.darpa.mil/
    - jcoffland 10y ago

frank_jaeger 10y ago

>A complex simulation with many long-lived agents. No need to start a new one here, just download Dwarf Fortress.

outworlder 10y ago
Except that DF Dwarves are, by definition, not long-lived.

GuiA 10y ago

"If you do not work on an important problem, it's unlikely you'll do important work. It's perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, `important problem' must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It's not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important." Richard Hamming, "You and Your Resarch" http://www.cs.virginia.edu/~robins/YouAndYourResearch.html http://www.cs.virginia.edu/~robins/YouAndYourResearch.html

jcoffland 10y ago
When you are uber rich and have no real technical skills you think that things get done by rich people demanding it and throwing money at it.
- jackmott 10y ago
  Well, that can help.
pavlov 10y ago
If you do not work on an important problem, it's unlikely you'll do important work. An anecdotal counterpoint would be that Einstein was working as a patent examiner in 1905, his "annus mirabilis"... (Of course it's not very useful to generalise Einstein's career.)
- nostrademons 10y ago
  I thought that the point the grandparent is highlighting is that "important work" is not defined by what other people think is important, but by the fact that you have a unique angle of attack that suddenly makes a previously-unsolved problem tractable. Einstein had a unique angle of attack for all 4 of his annus mirabilis papers, and the photoelectric effect, Michelson-Morley, and Brownian motion were all known to be important unsolved paradoxes in the physics of the day.

sozforex 10y ago

#1 - reminded me of "Endgame: Singularity" (https://en.wikipedia.org/wiki/Endgame%3A_Singularity https://en.wikipedia.org/wiki/Endgame%3A_Singularity)

deutronium 10y ago

The security project reminded me of the DARPA Cyber Grand Challenge - https://www.cybergrandchallenge.com/ https://www.cybergrandchallenge.com/

superswordfish 10y ago

> 2. Build an agent to win online programming competitions. A program that can write other programs would be, for obvious reasons, very powerful. The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you. Software engineers need to recognize this common threat and organize (labor) sooner than later. Even more important is for all engineers to plan for an imminent future where developers are not paid like they are now, if at all. We have it good, but we will be automated away like everybody else, just a little later.

riboflava 10y ago
The solution to everyone being automated away isn't to halt the advances of automation and inflict inefficient make-work on the economy, it's to get used to the idea that we will all become effectively retirees.
aluhut 10y ago
Seriously, why would you need programmers if the machine does it better (this needs to be assessed somehow first)?
city41 10y ago
Why should we fight that? That's no different from taxi companies fighting Uber, or how Intuit lobbies congress to avoid automatic filing of taxes. Hampering progress for the greater good just because it harms you in the short term is never a good idea.
- superswordfish 10y ago
  This wouldn't harm me in the short term alone, it would be a lost career and a life catastrophe. A taxi driver can work as an Uber driver. I cannot become an automated programmer. So I don't give a damn about the "greater good" in this context, and question that premise also: Sibling mentions getting used to everybody being retirees, but what I see is collective poverty with token conveniences. Time will tell, etc.
  - 10y ago

vonnik 10y ago

Riffing on project no. 1, here's our post on "How to Control AI": http://deeplearning4j.org/skynet http://deeplearning4j.org/skynet

freyir 10y ago

The goals are an AI that can augment itself (project 2) and defend itself (project 3). I've found Musk's fear of AI to be a little overblown, but this is nuclear weapon type rationalizing: "AI could be end the world as we know it, so we'd better race to create it first." It's interesting that Projects 2 and 3 would make great businesses even if they only got halfway to "existential threat" level AI. Then there's project 4, version 0.1 of Musk's "we're all living inside a simulation" claim. Musk as creator of worlds.

tom_b 10y ago

#2 - Build an agent to win online programming competitions. Given that online programming competitions tend to fall into distinct classes (dynamic programming, algorithmic challenges, graph problems, string problems), this seems maybe more "solvable" with a non-AI implementation? Imagine you have a framework that can spit out sub-pieces of a solution that worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected components in each graph | spit answer of graph with lowest number of SCC). Then you need to grind out some type of expert system to replicate the competitive programmer who currently chunks together those framework pieces. Of course, this probably doesn't work at all for something like kaggle or stockfighter.io, where parsing the instructions and observing a dynamic system are key parts of the hacking. I am more thinking of SPOJ or similar . . .

cynicaldevil 10y ago
The hardest part here is not the actual generation of code, but to analyze the problem first and to determine what data structures/algorithms are best suited to tackle it. Also, there are a lot of real world examples used in these problems, so the AI would need a general understanding of the modern-day world, which again, is hard. Lastly, even if the AI figures out the implementation, it has to take care of all the edge cases too, so that the test cases pass. But I suppose this is a trivial problem compared to the other two I mentioned above...
chriswarbo 10y ago
> non-AI implementation > a framework that can spit out sub-pieces of a solution > some type of expert system What you've described is an AI implementation.
nl 10y ago
There's been significant progress in data science competitions (Kaggle etc). Not so much parsing of the problem itself, but building generic frameworks where - given data and a target variable - it will work out the type of problem (eg classification vs regression) and develop a predictive model without human intervention. ICML has had workshops for the last two years on this problem[1][2] [1] https://sites.google.com/site/automl2016/ https://sites.google.com/site/automl2016/ [2] https://sites.google.com/site/automlwsicml15/ https://sites.google.com/site/automlwsicml15/

Cybiote 10y ago

#2 Is incredibly ambitious but I expect it should be doable by a lone grad student over the course of a summer. Something like this would range in complexity from completely redefining programming languages (drastically lowering the skill required for complex programming tasks and thus feeding into #1) all the way up to general intelligence. The tests referenced in [https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf], currently dominated by information retrieval techniques, seem more realistic and still feel hopelessly far away. #3 should also allow looking into the use of AI to break into systems, in addition to detecting and defending against AI breaking into systems. A prototype for #4 could create an environment where trilobyte fuzzers could co-evolve into fearsome Artificial intelligences. The AIs that break into systems need not be as smart as the systems being broken into. Just as viruses are much less complicated than eukaryotic cells and yet are capable of wreaking great havoc against mammals, might this be a possible mad deterrent against out of control AI, developed by #1's opponents who use #2's breakthroughs? No AI could plausibly be bug free. See also: Schild's Ladder. #4 It would also be cool if humans were allowed to visit and interact with this virtual world. See also: The Lifecycle of Software Objects These projects have the same feel as: A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE, whose #4 and #1[+] were the only ones to see much progress. It's difficult to say whether we are 10 or another 50 years away from making meaningful progress on OpenAI's list, but I'm glad they made it because it seems somewhere along the past 60 years, we forgot how to dream. [+] #3 warrants a honorable mention.

hooo 10y ago
You seriously think #2 is so easy a lone grad student could casually do it over a summer? ......
- skizm 10y ago
  It was tongue in cheek. The original AI researchers posed the problem of General Intelligence to a single grad student and thought he would have it completed by the end of the summer... 50+ years later here we are with no solution.
10y ago

akeck 10y ago

I christen number 1 "The Ghost in the Shell Problem", in reference to the project at the core of the first movie's plot.

jesalg 10y ago

This reminds me of an excellent two post series by Tim Urban about the upcoming AI revolution: http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...

vonnik 10y ago

I'm acutely curious how we can attack the problem of detecting undisclosed AI breakthroughs, particularly those made by organizations uninterested in ever revealing them. Commercially, yes, many organizations want to boast about their AI, and we may find those AIs via games or news. But many organizations involved in national security and financial markets will want to hide their advantage. I would love to hear more about how Open AI thinks this problem can be approached.

throwwit 10y ago
This is the biggest problem IMO. Just imagine a couple of AIs that systematically discover a few exploitable loopholes or trading algos all at once; it would be like black & scholes exponentiated. I doubt outside research would be able to infer anything from available data.
blahi 10y ago
Step one: Stop calling equations "AI".
- Bjartr 10y ago
  Step zero: Define "Intelligence"consistently and unambiguously.
crypto5 10y ago
By analyzing sequences of amazingly brilliant developments/products/inventions/movements performed by single company or conglomerate of linked companies, and which are few levels above what competitors can deliver. I am not from OpenAI though.
- vonnik 10y ago
  By that measure, startups are powered by AI and large corps are not. By which I mean, so many variables go into successful product development. Google arguably has the strongest AI in the world, but it has been remarkably bad at introducing new products recently, so those two phenomena seem decoupled to me...

auggierose 10y ago

That stuff sounds so generic, not sure what is special about it :-)

feral 10y ago

> Detect if someone is using a covert breakthrough AI system in the world. I bet the real purpose of this is not "It seems important to detect this", at least against superhuman AGI. As that's asking you to win a game against a more intelligent game player, who is going to be able to out-think you and hide. It seems more likely that: The real purpose of this problem is to impose a 'secrecy tax' on entities who would like to recoup investment by covertly using such a system, thus reducing the incentive to build one in secret. I.e. if you know someone has invested in research that's going to limit your ability to profit from covertly building an AGI, this means you can only use it up to the limit of game theoretic detectability, reducing the profit motive of acting covertly in the first place. Its a bit like once you break Enigma, you have to be careful how you use it. Having alternative explanations for how you got your edge then becomes valuable. So, if you see a corporation building something that could provide an alternative explanations for huge returns, other than AGI - ideally for something that's very noisy (high variance) - and where it'll be hard to tell whether their success was by chance or on purpose, that's probably (very weak) evidence (in the bayesian sense) they are working on a secret AGI. I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill.

mindcrime 10y ago
I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill. Maybe pg is actually an artificial super-intelligence. I mean, has anybody actually seen him in real life???

thuruv 10y ago

This will certainly backfire

nerdy 10y ago

Programs writing other programs is an interesting concept, but isn't it naive in most cases? If program X can write program Y, program X is probably unnecessarily complex with regards to solving problem Y. Program X could simply execute program Y's operations at their abstract level, which would save it all of the outputted language understanding and translation. The only tangible difference between the two would be the ability to "save" the commands, at which point program X is a scripting language and the input commands are the script. Wouldn't this solution be better in most cases? If program X at runtime requires significant configuration to produce program Y but those configuration inputs are not saved. Then if program Y were to be changed, program X must be rerun with all of the configuration input again. In the scripting language scenario, the script is modified. For a programming language to receive something simpler than the language it is written in, the ideas must be abstracted to reduce the amount of code required to perform the task. For example simple addition of multiple elements can be reduced to a "sum" function. The "programmer" using this must still have knowledge of the sum function and its use. Abstractions require the system to have knowledge of that domain, so with each new domain a whole other set of abstractions and complications are introduced. So if a machine were to write the entire program, how does it differ from machine learning? While machine learning might be often used for smaller algorithms, this case would apply ML to the entire problem as sets of smaller problems. If a human must still program in some abstracted sense, isn't it just a scripting/programming language? How would a component of this suggested project not fit into the "machine learning" or "scripting/programming language" categories?

chewxy 10y ago

Anyone who starts a project for #1 will get instantly targeted by Samaritan's agents, right?

swah 10y ago

Is this job application for humans only or AI can apply?