5 ms·

An important caveat from the paper >Moreover, we follow previous work in accelerating block breaking because learning to hold a button for hundreds of consecut

An important caveat from the paper >Moreover, we follow previous work in accelerating block breaking because learning to hold a button for hundreds of consecutive steps would be infeasible for stochastic policies, allowing us to focus on the essential challenges inherent in Minecraft.

JohnKemeny 2y ago
I'm not sure it's a serious caveat if the "hint" or "control" is in the manual.
- suddenlybananas 2y ago
  Sorry, I don't quite follow what you mean?
  - franktankbank 2y ago
    I didn't read the manual and when I was trying to help my kid play the game I couldn't figure out how to break blocks.
thesz 2y ago
"It allows AI to understand its physical environment and also to self-improve over time, without a human having to tell it exactly what to do."
- ks1723 2y ago
  I my view, the 'exactly' is crucial here. They do implicitly tell the model what to do by encoding it in the reward function: In Minecraft, the team used a protocol that gave Dreamer a ‘plus one’ reward every time it completed one of 12 progressive steps involved in diamond collection — including creating planks and a furnace, mining iron and forging an iron pickaxe. This is also why I think the title of the article is slightly misleading.
  - wongarsu 2y ago
    It's kind of fair, humans also get rewarded for those steps when they learn Minecraft
    - xwolfi 2y ago
      But they don't learn that way at all, my 7yo learns by watching youtubers. There's a whole network of people teaching each other the game, that's almost more fun than playing it alone.
kharak 2y ago
In my mind, this generalizes to the same problem with other non-stochastic (deterministic) operations like logical conclusions (A => B) . I have a running bet with friend that humans encode deterministic operations in neural networks, too, while he thinks there has to be another process at play. But there might be something extra helping our neural networks learn the strong weights required for it. Or the answer is again: "more data".
toxik 2y ago
Like all things RL, it is 99.9% about engineering the environment and rewards. As one of the authors stated elsewhere here, there is a reward for completing each of 12 steps necessary to find diamonds. Mostly I'm tired of RL work being oversold by its authors and proponents by anthropomorphizing its behaviors. All while this "agent" cannot reliably learn to hold down a button, literally the most basic interaction of the game.
- red75prime 2y ago
  The "no free lunch" theorem. You can't start from scratch and expect your program to repeat 4 billion years of evolution collecting inductive biases useful in our corner of our Universe in a matter of hours[1]. While it's possible to bake in this particular inductive bias (repetitive actions might be useful), they decided not to (it's just not that interesting). [1] And you certainly can't reproduce the observation selection effect in a laboratory. That is the thing that makes it possible to overcome the "no free lunch" theorem: our existence and intelligence are conditional on evolution being possible and finding the right biases. We have to bake in inductive biases to get results. We have to incentivize behaviors useful (or interesting) to us to get useful results instead of generic exploration.
  - kypro 2y ago
    > You can't start from scratch and expect your program to repeat 4 billion years of evolution collecting inductive biases useful in our corner of our Universe in a matter of hours Really? Minecraft's gameplay dynamic are not particularly complex... The AI here isn't learning highly complex rules about the nuances of human interaction or learning to detect the relatively subtle differences between various four legged creatures based on small differences in body morphology. In these cases I could see how millions of years of evolution is important to at least give us and other animals a head start when entering the world. If the AI had to do something like this to progress in Minecraft then I'd get why learning those complexities would be skipped over. But in this case a human would quickly understand that holding a button creates a state which tapping a button does not, and therefore would assume this state could be useful to explore further states. Identifying this doesn't seem particularly complex to me. If the argument is that it will take slightly longer for an AI to learn patterns in dependant states then okay, sure, but I think arguing that learning that holding a button creates a new state is such a complex problem that we couldn't possibly expect an AI to learn it from scratch within a short timeframe is a very weak argument. It's just not that complex. To me this suggests that current algorithms are lacking.
Hamuko 2y ago
Turns out that AI are much better at playing video games if they're allowed to cheat.
lgeorget 2y ago
Well, to be fair... I (a human) had to look it up online the first time I played as well. I was repeatedly clicking on the same tree for an entire minute before that. I even tried several different trees just in case.
- fusionadvocate 2y ago
  But it is possible to discover by holding down the button and realizing the block is getting progressively more "scratched".
FrustratedMonky 2y ago
"accelerating block breaking because learning to hold a button for hundreds of consecutive steps " This is fine, and does not impact the importance of figuring out the steps. Anybody that has done any tuning on systems that run at different speeds, the adjusting for the speed difference is just engineering, and allows you to get on with more important/inventive work.
o11c 2y ago
And a relevant piece of ancient wisdom (exact date not known, but presumably before 1970): > In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. > “What are you doing?”, asked Minsky. > “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. > “Why is the net wired randomly?”, asked Minsky. > “I do not want it to have any preconceptions of how to play”, Sussman said. > Minsky then shut his eyes. > “Why do you close your eyes?”, Sussman asked his teacher. > “So that the room will be empty.” > At that moment, Sussman was enlightened.

JohnKemeny 2y ago

I'm not sure it's a serious caveat if the "hint" or "control" is in the manual.

suddenlybananas 2y ago
Sorry, I don't quite follow what you mean?
- franktankbank 2y ago
  I didn't read the manual and when I was trying to help my kid play the game I couldn't figure out how to break blocks.

thesz 2y ago

"It allows AI to understand its physical environment and also to self-improve over time, without a human having to tell it exactly what to do."

ks1723 2y ago
I my view, the 'exactly' is crucial here. They do implicitly tell the model what to do by encoding it in the reward function: In Minecraft, the team used a protocol that gave Dreamer a ‘plus one’ reward every time it completed one of 12 progressive steps involved in diamond collection — including creating planks and a furnace, mining iron and forging an iron pickaxe. This is also why I think the title of the article is slightly misleading.
- wongarsu 2y ago
  It's kind of fair, humans also get rewarded for those steps when they learn Minecraft
  - xwolfi 2y ago
    But they don't learn that way at all, my 7yo learns by watching youtubers. There's a whole network of people teaching each other the game, that's almost more fun than playing it alone.

kharak 2y ago

In my mind, this generalizes to the same problem with other non-stochastic (deterministic) operations like logical conclusions (A => B) . I have a running bet with friend that humans encode deterministic operations in neural networks, too, while he thinks there has to be another process at play. But there might be something extra helping our neural networks learn the strong weights required for it. Or the answer is again: "more data".

toxik 2y ago

Like all things RL, it is 99.9% about engineering the environment and rewards. As one of the authors stated elsewhere here, there is a reward for completing each of 12 steps necessary to find diamonds. Mostly I'm tired of RL work being oversold by its authors and proponents by anthropomorphizing its behaviors. All while this "agent" cannot reliably learn to hold down a button, literally the most basic interaction of the game.

red75prime 2y ago
The "no free lunch" theorem. You can't start from scratch and expect your program to repeat 4 billion years of evolution collecting inductive biases useful in our corner of our Universe in a matter of hours[1]. While it's possible to bake in this particular inductive bias (repetitive actions might be useful), they decided not to (it's just not that interesting). [1] And you certainly can't reproduce the observation selection effect in a laboratory. That is the thing that makes it possible to overcome the "no free lunch" theorem: our existence and intelligence are conditional on evolution being possible and finding the right biases. We have to bake in inductive biases to get results. We have to incentivize behaviors useful (or interesting) to us to get useful results instead of generic exploration.
- kypro 2y ago
  > You can't start from scratch and expect your program to repeat 4 billion years of evolution collecting inductive biases useful in our corner of our Universe in a matter of hours Really? Minecraft's gameplay dynamic are not particularly complex... The AI here isn't learning highly complex rules about the nuances of human interaction or learning to detect the relatively subtle differences between various four legged creatures based on small differences in body morphology. In these cases I could see how millions of years of evolution is important to at least give us and other animals a head start when entering the world. If the AI had to do something like this to progress in Minecraft then I'd get why learning those complexities would be skipped over. But in this case a human would quickly understand that holding a button creates a state which tapping a button does not, and therefore would assume this state could be useful to explore further states. Identifying this doesn't seem particularly complex to me. If the argument is that it will take slightly longer for an AI to learn patterns in dependant states then okay, sure, but I think arguing that learning that holding a button creates a new state is such a complex problem that we couldn't possibly expect an AI to learn it from scratch within a short timeframe is a very weak argument. It's just not that complex. To me this suggests that current algorithms are lacking.

Hamuko 2y ago

Turns out that AI are much better at playing video games if they're allowed to cheat.

lgeorget 2y ago

Well, to be fair... I (a human) had to look it up online the first time I played as well. I was repeatedly clicking on the same tree for an entire minute before that. I even tried several different trees just in case.

fusionadvocate 2y ago
But it is possible to discover by holding down the button and realizing the block is getting progressively more "scratched".

FrustratedMonky 2y ago

"accelerating block breaking because learning to hold a button for hundreds of consecutive steps " This is fine, and does not impact the importance of figuring out the steps. Anybody that has done any tuning on systems that run at different speeds, the adjusting for the speed difference is just engineering, and allows you to get on with more important/inventive work.

o11c 2y ago

And a relevant piece of ancient wisdom (exact date not known, but presumably before 1970): > In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. > “What are you doing?”, asked Minsky. > “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. > “Why is the net wired randomly?”, asked Minsky. > “I do not want it to have any preconceptions of how to play”, Sussman said. > Minsky then shut his eyes. > “Why do you close your eyes?”, Sussman asked his teacher. > “So that the room will be empty.” > At that moment, Sussman was enlightened.