A couple questions I have after skimming the paper, so forgive me if they were answered somewhere in the 54 page manuscript: They mention in A.3 that they expl

A couple questions I have after skimming the paper, so forgive me if they were answered somewhere in the 54 page manuscript: They mention in A.3 that they explicitly reject dynamically generated training worlds/games that collide with their evaluation sets, but do they ensure that dynamic training games are sufficiently "distant" from their evaluation sets regardless of whether or not there's a direct collision? If not, you might still end up training on something quite similar to your test dataset. Figure 27 kind of suggests that might happen for some games given that the vast majority of the held out games have relatively poor transfer performance but a few are really good. Speaking of Figure 27, while the reward looks good it would have been really nice to show some examples of what these "zero-shot" games look like versus the fine tuned version. Is the gap in the reward between the raw vs fine tuned version significant? Wouldn't we expect the internal state representation to be more definitive in classifying the state of the agent during the simulation as the agent moves around the environment? From their examples: Figure 20,21, and 22 it almost looks like it either flags the state as "early" or "success." Not sure we're getting the expected performance out of it.