36 ms·
> An "exploration-based reward function" wouldn't be enough. Because this would assume that exploration has (more or less) deterministic outcomes. But given the
by eru 3y ago
> An "exploration-based reward function" wouldn't be enough. Because this would assume that exploration has (more or less) deterministic outcomes. But given the dice in Nethack it actually does not! You can do "everything right" and still die in almost 50% of the cases. How to infer any meaningful "world model" from such events? Imho you can't.
You can actually do that very well. When they train LLM on text, the same prefix doesn't always lead to the same next token. And they handle that just fine.
Btw, Nethack isn't actually random: the dice use a pretty broken PRNG, and 'luck manipulation' is a thing. A computer might not actually care about the difference between spoiler-y tactics that are legible to humans, and PRNG manipulation.
(The state of the PRNG is a relatively small number of bits. Various actions can advance the PRNG, without causing any other change in the world. So you can basically make sure that you are always maximally lucky, if you can somehow recover the hidden PRNG state from the output of the program, and then model it.)