11 ms·

Why I’m Remaking OpenAI Universe

224 points by evc123 9y ago

evc123 9y ago
https://github.com/unixpickle/muniverse https://github.com/unixpickle/muniverse https://github.com/unixpickle/demoverse https://github.com/unixpickle/demoverse
dswalter 9y ago
I'm a little surprised, but this seems like a good idea. HTML5 certainly has a brighter present and future than flash, and skipping the OCR stem should save quite a few CPU cycles.
misiti3780 9y ago
Did openAI really unofficially abandoned universe ?
- toisanji 9y ago
  yes
  - adewinter 9y ago
    They switched over to OpenAI Gym which is much broader in scope (able to play Steam based video games).
    - aerovistae 9y ago
      This question on Quora contradicts you, now I don't know who to believe... https://www.quora.com/What-is-the-difference-between-OpenAIs-Universe-and-Gym https://www.quora.com/What-is-the-difference-between-OpenAIs...
- aerovistae 9y ago
  Yeah, really interested in hearing their take on this. It's not often you see a Musk-sponsored enterprise cast a major project aside without public comment.
  - chronic61a 9y ago
    It's because the people actually working on AI, including OpenAI, finally knocked some sense into Elon Musk. He finally realized how far behind AI is (it is a glorified linear regression) and we won't be seeing general AI for at least another 40 years. Source: Am an AI research scientist.
    - computerex 9y ago
      Would be interested to know how you reached that 40 years number. I don't think we are even remotely close to AGI, 40 years to me seems extremely optimistic. That's within my lifetime.
      - adrianN 9y ago
        They're a scientist: https://xkcd.com/678/ https://xkcd.com/678/
        eli_gottlieb 9y ago
        That's a little sadder now that I've had the "fourth quarter next year" thing happen to me personally.
      - ewjordan 9y ago
        Probably the same way everyone does, by pulling it out of thin air as a guess. When nobody even knows what theoretical breakthroughs are necessary, you'll always end up with a scattershot all over the place, even amongst experts. Try asking working mathematicians how long until the Riemann hypothesis is resolved one way or another, or look at what people were saying about Fermat's Last Theorem up until it was solved. What we do know is that current techniques won't get us close to AGI, so something new is needed (or perhaps like backprop, something old will work once we have enough compute power). Personally I'm bullish on AGI because I have strikingly low faith in the ability of evolution to operate very effectively as a tool for algorithm discovery, so I suspect that once we've hit the compute threshold we'll find that many different algorithms can do the trick, and 40 years is probably not out of the question for us to hit that point (or 10, or 100), depending who you talk to about what the compute threshold might be. I'd caution against putting too much weight in what experts say, though, since with a tiny few set of exceptions anyone working on "AI" today is actually just working on narrow AI, which is, as someone put it, just glorified linear regression. Those tools will almost certainly be part of the solution, but only in the sense that the classical theory of Diophantine equations was part of Weil's proof of Fermat's Last Theorem - they are not the core of the theoretical approach.
deleted 9y ago
[deleted]
zach417 9y ago
I echo all of your issues with running Universe. I have a decrepit Macbook, and it was actually not possible for me to use it at all.
- forgotmyhnacc 9y ago
  If you have trouble running universe, how are you going to run RL algorithms that use lots of gpu and CPU?
gdb 9y ago
(I work at OpenAI.) Great project. We've found that the VNC Universe environments are hard for today's RL algorithms primarily due to the their async nature. We're currently working on a new set of Universe environments without VNC; I'm very happy to see others inspired by the core ideas of Universe as well.
- Aqueous 9y ago
  It seems like you might be duplicating work? At the end he mentions he's dropping VNC in favor of headless Chrome.
- evc123 9y ago
  recruit Alex Nichol (unixpickle).
- du_bing 9y ago
  Oh, Greg, nice to see you here, I am eager to see some solid Universe environments without VNC, that will be interesting.
- unixpickle 9y ago
  (Author here). Hi Greg! I am excited to hear about the new Universe environments. I want as many RL environments as possible for my upcoming project, so I will probably draw from Universe and ALE as well as µniverse. I took a lot of inspiration from Universe and am grateful for OpenAI's work on RL in general :). I probably wouldn't have started on this project if a company like OpenAI hadn't already decided it was a worthy goal.
- pixelHD 9y ago
  honest question, how interested is the academia/industry in deep learning libraries & game engines integrations? I've worked on unreal and tensorflow the last semester, and I found out that there aren't any existing integrations. I will probably work on a plugin, but I wanted to know if there is any interest? The way I see it, having hooks into the engines themselves helps with what the article talks about - not needing to go through VNCs or other _glue_ to get realtime data. It could potentially send the framebuffers themselves directly from the game/simulation and tie in the actions back to the game/simulation. And using framebuffers is just one direction, we could instead stream the co-ords/the current payoff/etc. Also, having such plugins would help with the adoption in both directions - games now have an always updating/learning AI (might need a network connection + cloud backend), and researchers can have training/testing environments.
hackpert 9y ago
This is great. Using HTML5 games in a headless browser makes a lot of sense because the need for VNC is circumvented. However, I think that while OpenAI's implementation is certainly not the best, having access just the information on the screen is not a bad idea in itself as a (maybe optional) constraint. With access to the game's internal state we don't even need RL for solving a large number of games - algorithms like NEAT are sufficient.
- Houshalter 9y ago
  This project doesn't change that. The agents still only get screenshots of the game as far as I understand. However I think this approach is bad. Machine vision is a separate problem from reinforcement learning. You shouldn't need to be able to do both well. Machine vision consumes a ton of processing power and researcher time in figuring out the hyperparameters. And all it's doing is figuring out information that's already in memory like the location of various objects and the score. It really limits what can be done. E.g. the famous atari playing AIs by deepmind were limited to no memory and only knowing the last few frames, because backpropagating through thousands of frames was too expensive. Because of the way NNs work, it's trivial to separate out the machine vision into a separate module. So if you have a good RNN reinforcement learning system, you can easily add a machine vision learning system to it later if you need.
  - unixpickle 9y ago
    In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401 https://arxiv.org/abs/1606.03401.
strin 9y ago
Awesome project. Despite the flaws, the nice thing with VNC is its universality to support any apps on a computer. Using HTML5 in a browser limits the scope of things we could encapsulate as environments, and makes it less "universe". However, there is a difference between the universality of the tech stack and the exposed interface. In my opinion, the future universe would be rich clusters of RL environments with unified API, each of which implemented using different underlying technology to meet the desired synchronicity and frame performance. HTML5 could deliver one of such clusters.
- unixpickle 9y ago
  I'm pretty sure that was the goal of OpenAI Gym. Gym tries to provide a generic interface for RL environments, and imho it does a nice job. I am working on Python bindings for µniverse now, which should allow µniverse to integrate with Gym.
tomjacobs 9y ago
Missed opportunity for a Rick and Morty Microverse reference here as the name
Houshalter 9y ago
Why not use game emulators? With popular NES emulators you can advance the game frame by frame. You can read the raw memory addresses that correspond to the score. You can dump the memory at any time and reload the game to a specific game state. You can even manipulate the games in many fun ways by messing around with the game memory. Or give an AI algorithm access to memory addresses as additional information, instead of relying on pure machine vision, if you want to do that.. Here's an example of a guy who made a general game playing algorithm that brute forces it's way through any NES game: https://www.youtube.com/watch?v=xOCurBYI_gY https://www.youtube.com/watch?v=xOCurBYI_gY This isn't necessarily interesting from an AI perspective - the playing algorithm is just brute force. But it shows what can be done with the platform, easily reloading to previous states and exploring counterfactual futures (which is exactly the sort of thing RL algorithms do.) He also has a cool algorithm for finding the objective function of an arbitrary game, by watching a human play, and seeing what memory addresses increment. Which is a lot more easy to use than writing OCR code to read the score and game over states from the screen.
zzh8829 9y ago
I am also working on related project. Flash and HTML5 games in chrome are great but they are very far away from the initially promised full blown GTA5, Starcraft and other complex envs. I am in process of remaking the Universe framework for host machine, since running those computation intensive games at reasonable frame is nearly impossible inside docker or virtual machines.
namuol 9y ago
Funny, I have an old (unfinished) HTML5 space-exploration game by the same name: https://github.com/namuol/muniverse https://github.com/namuol/muniverse If I had more time I'd submit a PR to integrate it...
Cellestro 9y ago
Congratulations on the initiative, it looks very cool! Indeed, we found that running asynchronous environments, while possible, proved to be too cumbersome for research. We're now working on a synchronous set of environments for universe that are easier to use.
daveguy 9y ago
According to the author, "Universe never really took off in the AI world." That's a bit premature for a project that was just released less than 7 months ago, isn't it? https://blog.openai.com/universe/ https://blog.openai.com/universe/ Edit: that said the project seems to have some interesting and needed improvements (esp time adjustment). Glad to see dialog between muniverse and openai here.
make3 9y ago
I wonder what's happening with OpenAI. Most big names are leaving.
- fggh 9y ago
  Please elaborate...
  - make3 9y ago
    Well, Ian Goodfellow and Andrej Karpathy for starters

224 points by evc123 9y ago

https://github.com/unixpickle/muniverse https://github.com/unixpickle/muniverse https://github.com/unixpickle/demoverse https://github.com/unixpickle/demoverse

dswalter 9y ago

I'm a little surprised, but this seems like a good idea. HTML5 certainly has a brighter present and future than flash, and skipping the OCR stem should save quite a few CPU cycles.

misiti3780 9y ago

Did openAI really unofficially abandoned universe ?

toisanji 9y ago
yes
- adewinter 9y ago
  They switched over to OpenAI Gym which is much broader in scope (able to play Steam based video games).
  - aerovistae 9y ago
    This question on Quora contradicts you, now I don't know who to believe... https://www.quora.com/What-is-the-difference-between-OpenAIs-Universe-and-Gym https://www.quora.com/What-is-the-difference-between-OpenAIs...
aerovistae 9y ago
Yeah, really interested in hearing their take on this. It's not often you see a Musk-sponsored enterprise cast a major project aside without public comment.
- chronic61a 9y ago
  It's because the people actually working on AI, including OpenAI, finally knocked some sense into Elon Musk. He finally realized how far behind AI is (it is a glorified linear regression) and we won't be seeing general AI for at least another 40 years. Source: Am an AI research scientist.
  - computerex 9y ago
    Would be interested to know how you reached that 40 years number. I don't think we are even remotely close to AGI, 40 years to me seems extremely optimistic. That's within my lifetime.
    - adrianN 9y ago
      They're a scientist: https://xkcd.com/678/ https://xkcd.com/678/
      - eli_gottlieb 9y ago
        That's a little sadder now that I've had the "fourth quarter next year" thing happen to me personally.
    - ewjordan 9y ago
      Probably the same way everyone does, by pulling it out of thin air as a guess. When nobody even knows what theoretical breakthroughs are necessary, you'll always end up with a scattershot all over the place, even amongst experts. Try asking working mathematicians how long until the Riemann hypothesis is resolved one way or another, or look at what people were saying about Fermat's Last Theorem up until it was solved. What we do know is that current techniques won't get us close to AGI, so something new is needed (or perhaps like backprop, something old will work once we have enough compute power). Personally I'm bullish on AGI because I have strikingly low faith in the ability of evolution to operate very effectively as a tool for algorithm discovery, so I suspect that once we've hit the compute threshold we'll find that many different algorithms can do the trick, and 40 years is probably not out of the question for us to hit that point (or 10, or 100), depending who you talk to about what the compute threshold might be. I'd caution against putting too much weight in what experts say, though, since with a tiny few set of exceptions anyone working on "AI" today is actually just working on narrow AI, which is, as someone put it, just glorified linear regression. Those tools will almost certainly be part of the solution, but only in the sense that the classical theory of Diophantine equations was part of Weil's proof of Fermat's Last Theorem - they are not the core of the theoretical approach.

deleted 9y ago

[deleted]

zach417 9y ago

I echo all of your issues with running Universe. I have a decrepit Macbook, and it was actually not possible for me to use it at all.

forgotmyhnacc 9y ago
If you have trouble running universe, how are you going to run RL algorithms that use lots of gpu and CPU?

gdb 9y ago

(I work at OpenAI.) Great project. We've found that the VNC Universe environments are hard for today's RL algorithms primarily due to the their async nature. We're currently working on a new set of Universe environments without VNC; I'm very happy to see others inspired by the core ideas of Universe as well.

Aqueous 9y ago
It seems like you might be duplicating work? At the end he mentions he's dropping VNC in favor of headless Chrome.
evc123 9y ago
recruit Alex Nichol (unixpickle).
du_bing 9y ago
Oh, Greg, nice to see you here, I am eager to see some solid Universe environments without VNC, that will be interesting.
unixpickle 9y ago
(Author here). Hi Greg! I am excited to hear about the new Universe environments. I want as many RL environments as possible for my upcoming project, so I will probably draw from Universe and ALE as well as µniverse. I took a lot of inspiration from Universe and am grateful for OpenAI's work on RL in general :). I probably wouldn't have started on this project if a company like OpenAI hadn't already decided it was a worthy goal.
pixelHD 9y ago
honest question, how interested is the academia/industry in deep learning libraries & game engines integrations? I've worked on unreal and tensorflow the last semester, and I found out that there aren't any existing integrations. I will probably work on a plugin, but I wanted to know if there is any interest? The way I see it, having hooks into the engines themselves helps with what the article talks about - not needing to go through VNCs or other _glue_ to get realtime data. It could potentially send the framebuffers themselves directly from the game/simulation and tie in the actions back to the game/simulation. And using framebuffers is just one direction, we could instead stream the co-ords/the current payoff/etc. Also, having such plugins would help with the adoption in both directions - games now have an always updating/learning AI (might need a network connection + cloud backend), and researchers can have training/testing environments.

hackpert 9y ago

This is great. Using HTML5 games in a headless browser makes a lot of sense because the need for VNC is circumvented. However, I think that while OpenAI's implementation is certainly not the best, having access just the information on the screen is not a bad idea in itself as a (maybe optional) constraint. With access to the game's internal state we don't even need RL for solving a large number of games - algorithms like NEAT are sufficient.

Houshalter 9y ago
This project doesn't change that. The agents still only get screenshots of the game as far as I understand. However I think this approach is bad. Machine vision is a separate problem from reinforcement learning. You shouldn't need to be able to do both well. Machine vision consumes a ton of processing power and researcher time in figuring out the hyperparameters. And all it's doing is figuring out information that's already in memory like the location of various objects and the score. It really limits what can be done. E.g. the famous atari playing AIs by deepmind were limited to no memory and only knowing the last few frames, because backpropagating through thousands of frames was too expensive. Because of the way NNs work, it's trivial to separate out the machine vision into a separate module. So if you have a good RNN reinforcement learning system, you can easily add a machine vision learning system to it later if you need.
- unixpickle 9y ago
  In terms of "backpropagating through thousands of frames", it's not as expensive as you might think. I've used TRPO to train RNNs on games like Atari pong with thousands of frames per episode. This can be done via an algorithm that reduces the memory complexity of RNN backpropagation (these algorithms didn't exist in 2013). See for example https://arxiv.org/abs/1606.03401 https://arxiv.org/abs/1606.03401.

strin 9y ago

Awesome project. Despite the flaws, the nice thing with VNC is its universality to support any apps on a computer. Using HTML5 in a browser limits the scope of things we could encapsulate as environments, and makes it less "universe". However, there is a difference between the universality of the tech stack and the exposed interface. In my opinion, the future universe would be rich clusters of RL environments with unified API, each of which implemented using different underlying technology to meet the desired synchronicity and frame performance. HTML5 could deliver one of such clusters.

unixpickle 9y ago
I'm pretty sure that was the goal of OpenAI Gym. Gym tries to provide a generic interface for RL environments, and imho it does a nice job. I am working on Python bindings for µniverse now, which should allow µniverse to integrate with Gym.

tomjacobs 9y ago

Missed opportunity for a Rick and Morty Microverse reference here as the name

Houshalter 9y ago

Why not use game emulators? With popular NES emulators you can advance the game frame by frame. You can read the raw memory addresses that correspond to the score. You can dump the memory at any time and reload the game to a specific game state. You can even manipulate the games in many fun ways by messing around with the game memory. Or give an AI algorithm access to memory addresses as additional information, instead of relying on pure machine vision, if you want to do that.. Here's an example of a guy who made a general game playing algorithm that brute forces it's way through any NES game: https://www.youtube.com/watch?v=xOCurBYI_gY https://www.youtube.com/watch?v=xOCurBYI_gY This isn't necessarily interesting from an AI perspective - the playing algorithm is just brute force. But it shows what can be done with the platform, easily reloading to previous states and exploring counterfactual futures (which is exactly the sort of thing RL algorithms do.) He also has a cool algorithm for finding the objective function of an arbitrary game, by watching a human play, and seeing what memory addresses increment. Which is a lot more easy to use than writing OCR code to read the score and game over states from the screen.

zzh8829 9y ago

I am also working on related project. Flash and HTML5 games in chrome are great but they are very far away from the initially promised full blown GTA5, Starcraft and other complex envs. I am in process of remaking the Universe framework for host machine, since running those computation intensive games at reasonable frame is nearly impossible inside docker or virtual machines.

namuol 9y ago

Funny, I have an old (unfinished) HTML5 space-exploration game by the same name: https://github.com/namuol/muniverse https://github.com/namuol/muniverse If I had more time I'd submit a PR to integrate it...

Cellestro 9y ago

Congratulations on the initiative, it looks very cool! Indeed, we found that running asynchronous environments, while possible, proved to be too cumbersome for research. We're now working on a synchronous set of environments for universe that are easier to use.

daveguy 9y ago

According to the author, "Universe never really took off in the AI world." That's a bit premature for a project that was just released less than 7 months ago, isn't it? https://blog.openai.com/universe/ https://blog.openai.com/universe/ Edit: that said the project seems to have some interesting and needed improvements (esp time adjustment). Glad to see dialog between muniverse and openai here.

make3 9y ago

I wonder what's happening with OpenAI. Most big names are leaving.

fggh 9y ago
Please elaborate...
- make3 9y ago
  Well, Ian Goodfellow and Andrej Karpathy for starters