One important thing to note here is that this model was trained purely in a supervised fashion. It would be interesting to see a paper at a similar scale that's

One important thing to note here is that this model was trained purely in a supervised fashion. It would be interesting to see a paper at a similar scale that's based on reinforcement learning. The reinforcement learning context (specifically the exploring part) gives a lot more opportunities to see the effects of positive/negative transfer. That approach would of course be much more expensive, though.