The paper says that the input is video + the set of sprites used in each game. This simplifies the problem quite a bit. I tried to work from screenshots of Mari

The paper says that the input is video + the set of sprites used in each game. This simplifies the problem quite a bit. I tried to work from screenshots of Mario to recover the sprites, and it was a surprisingly challenging problem.