3 ms·

Well if it was only trained on non-full moon days, even if the model did detect a difference it would have no idea how to adapt its play style.

by WJW 2y ago

jsjohnst 2y ago
As another commenter said, it’s quite obvious about it. For such a key difference, I’d think they’d have the model run record a log for later inspection. Or they’d be watching the game play. Or something similar.
- WJW 2y ago
  If the model was trained over (say) a week during there was no full moon, how would the model know what this unexpected message means? It would probably just ignore it and continue playing as normal. Nethack is full of messages that can be safely ignored, so just another one would not be unusual. I don't agree that the creators would be watching the game play either. Usually during such training phases you'd run as many copies of the game as the available hardware can manage. I wouldn't be surprised if they had at least hundreds of runs going in parallel and the researcher is definitely not going to be watching them all. If anything, they are going to bed and train the model overnight as much as possible.
  - jsjohnst 2y ago
    > how would the model know what this unexpected message means? It would probably just ignore it and continue playing as normal. Then that’s a poor model then. Significant anomalies should be flagged for manual review, otherwise you corrupt datasets unintentionally.
thaumasiotes 2y ago
According to the article, the effect is that actions have better outcomes than they otherwise would. Assuming you don't adapt your playstyle in any way, how would that lead to worse overall performance?
- pixl97 2y ago
  Not really a nethack player, but in the comments on the article a player states this >edit: aha, I think this is it - attacked werecreatures are much more likely to summon help on full moons. Poor bot probably got overrun. It may have better outcomes in most situations, but if you're depending heavily on a particular strategy and that changes then you're in trouble.

Well if it was only trained on non-full moon days, even if the model did detect a difference it would have no idea how to adapt its play style.

by WJW 2y ago

Well if it was only trained on non-full moon days, even if the model did detect a difference it would have no idea how to adapt its play style.

jsjohnst 2y ago

As another commenter said, it’s quite obvious about it. For such a key difference, I’d think they’d have the model run record a log for later inspection. Or they’d be watching the game play. Or something similar.

WJW 2y ago
If the model was trained over (say) a week during there was no full moon, how would the model know what this unexpected message means? It would probably just ignore it and continue playing as normal. Nethack is full of messages that can be safely ignored, so just another one would not be unusual. I don't agree that the creators would be watching the game play either. Usually during such training phases you'd run as many copies of the game as the available hardware can manage. I wouldn't be surprised if they had at least hundreds of runs going in parallel and the researcher is definitely not going to be watching them all. If anything, they are going to bed and train the model overnight as much as possible.
- jsjohnst 2y ago
  > how would the model know what this unexpected message means? It would probably just ignore it and continue playing as normal. Then that’s a poor model then. Significant anomalies should be flagged for manual review, otherwise you corrupt datasets unintentionally.

thaumasiotes 2y ago

According to the article, the effect is that actions have better outcomes than they otherwise would. Assuming you don't adapt your playstyle in any way, how would that lead to worse overall performance?

pixl97 2y ago
Not really a nethack player, but in the comments on the article a player states this >edit: aha, I think this is it - attacked werecreatures are much more likely to summon help on full moons. Poor bot probably got overrun. It may have better outcomes in most situations, but if you're depending heavily on a particular strategy and that changes then you're in trouble.