4 ms·

Honestly, turning a supervised learning problem into a reinforcement learning problem is an extremely wasteful way to burn a lot of electricity. Supervised lea

by 317070 6y ago

Honestly, turning a supervised learning problem into a reinforcement learning problem is an extremely wasteful way to burn a lot of electricity. Supervised learning is orders of magnitude more efficient and better understood than reinforcement learning. Reinforcement learning is an algorithm that can help you find unknown solutions to interact in an optimal way with a system. To predict the next board state, you know the solution already. So, this is a supervised learning problem.

sillysaurusx 6y ago
If that were true, there wouldn't be any reason to write this submission's article. But the article points out that it's not merely a supervised learning problem; the number of board states for a 768x768 board is 768x768xN, where N is the number of time steps. How would you gather labels for every possible board state? It would require 768 factorial labels to represent every possibility. Consider the problem of face completion: https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_multioutput_face_completion.html#sphx-glr-auto-examples-miscellaneous-plot-multioutput-face-completion-py https://scikit-learn.org/stable/auto_examples/miscellaneous/... Given a set of faces, mask off random parts of the face, then try to fill in the holes. Neural nets excel at that kind of task, because as you say, it's a supervised learning problem: you "know the solution" (the missing face) and you can penalize / reward the model so that it learns a correct face on average. But game of life is nothing like that. Every board is a valid board. You can set the board to whatever values you want, and then say "Go!" and game of life will happily simulate it. Therefore the neural network's goal isn't to learn the board state; it's to learn the rules, and to obey the rules. Hence, self-play seems like a reasonable formulation. Without self-play, your learned network will always overfit or underfit different scenarios. E.g. even if it does well in general, a sparse board might throw it off. Or a board with a bunch of pieces in one corner. Or a board in a checkerboard pattern. But self-play will automatically handle all of these cases, because it's forced to learn the patterns over time. Minesweeper is another example: it's possible to win every game using nothing but logic (disregarding tiebreakers), so therefore the solution is known. But I don't think it can be formulated as a supervised learning problem. There are too many board states, and the board states are related to each other via logic ("am I obeying the game state?") rather than atoms ("what does a human face look like?")
- 317070 6y ago
  I can try to address some of your arguments one by one: > How would you gather labels for every possible board state? It would require 768 factorial labels to represent every possibility. You don't need to represent every possibility. Just a sufficiently large dataset and a neural network with a sufficient amount of prior and size. Note that e.g. video continuation networks already exist and deals with comparable data sizes. > But game of life is nothing like that. Every board is a valid board. Yes, but given the previous board (or the board N steps ago) there is only one valid board. Hence, a supervised learning problem. > Without self-play, your learned network will always overfit or underfit different scenarios. This confuses me. Why do you think self-play avoids over- and underfitting? How could one approach even solve both over- and underfitting at the same time? > Minesweeper is another example: it's possible to win every game using nothing but logic (disregarding tiebreakers), so therefore the solution is known. But I don't think it can be formulated as a supervised learning problem. Sure it can. If the solution is known, just train a neural network to mimic the solution. That's what network distillation is all about.
  - Agentlien 6y ago
    I've never heard of video continuation networks before and I would love to learn about it. Can you provide a link?

by 317070 6y ago

sillysaurusx 6y ago

If that were true, there wouldn't be any reason to write this submission's article. But the article points out that it's not merely a supervised learning problem; the number of board states for a 768x768 board is 768x768xN, where N is the number of time steps. How would you gather labels for every possible board state? It would require 768 factorial labels to represent every possibility. Consider the problem of face completion: https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_multioutput_face_completion.html#sphx-glr-auto-examples-miscellaneous-plot-multioutput-face-completion-py https://scikit-learn.org/stable/auto_examples/miscellaneous/... Given a set of faces, mask off random parts of the face, then try to fill in the holes. Neural nets excel at that kind of task, because as you say, it's a supervised learning problem: you "know the solution" (the missing face) and you can penalize / reward the model so that it learns a correct face on average. But game of life is nothing like that. Every board is a valid board. You can set the board to whatever values you want, and then say "Go!" and game of life will happily simulate it. Therefore the neural network's goal isn't to learn the board state; it's to learn the rules, and to obey the rules. Hence, self-play seems like a reasonable formulation. Without self-play, your learned network will always overfit or underfit different scenarios. E.g. even if it does well in general, a sparse board might throw it off. Or a board with a bunch of pieces in one corner. Or a board in a checkerboard pattern. But self-play will automatically handle all of these cases, because it's forced to learn the patterns over time. Minesweeper is another example: it's possible to win every game using nothing but logic (disregarding tiebreakers), so therefore the solution is known. But I don't think it can be formulated as a supervised learning problem. There are too many board states, and the board states are related to each other via logic ("am I obeying the game state?") rather than atoms ("what does a human face look like?")

317070 6y ago
I can try to address some of your arguments one by one: > How would you gather labels for every possible board state? It would require 768 factorial labels to represent every possibility. You don't need to represent every possibility. Just a sufficiently large dataset and a neural network with a sufficient amount of prior and size. Note that e.g. video continuation networks already exist and deals with comparable data sizes. > But game of life is nothing like that. Every board is a valid board. Yes, but given the previous board (or the board N steps ago) there is only one valid board. Hence, a supervised learning problem. > Without self-play, your learned network will always overfit or underfit different scenarios. This confuses me. Why do you think self-play avoids over- and underfitting? How could one approach even solve both over- and underfitting at the same time? > Minesweeper is another example: it's possible to win every game using nothing but logic (disregarding tiebreakers), so therefore the solution is known. But I don't think it can be formulated as a supervised learning problem. Sure it can. If the solution is known, just train a neural network to mimic the solution. That's what network distillation is all about.
- Agentlien 6y ago
  I've never heard of video continuation networks before and I would love to learn about it. Can you provide a link?